[Starlingx-discuss] How to turn off fault management?

[MacDonald, Eric]

Yes, SM can request mtce to reboot a controller and there will be an explicit mtce log for that (below)

controller-? is being force failed by SM

From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
Sent: Monday, June 24, 2019 7:48 AM
To: MacDonald, Eric; Liu, Tao; Ezpeer Chen; starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] How to turn off fault management?
Importance: High

Hi Eric,
I guess you mean sm should reboot a host, not shutdown. Is it correct?
But per Ezpeer’s description, it seems like shutdown.
BTW, it is with STX 1.0, from 2010.10 release branch.

Best Regards
Shuicheng

From: MacDonald, Eric [mailto:Eric.MacDonald at windriver.com]
Sent: Monday, June 24, 2019 7:40 PM
To: Lin, Shuicheng <shuicheng.lin at intel.com>; Liu, Tao <Tao.Liu at windriver.com>; Ezpeer Chen <ezpeerchen at gmail.com>; starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] How to turn off fault management?

SM does not power off a host.
When you say shutdown you mean power off correct ?

From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
Sent: Monday, June 24, 2019 7:38 AM
To: MacDonald, Eric; Liu, Tao; Ezpeer Chen; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] How to turn off fault management?
Importance: High

Hi Eric,
Here is the collect log for the issue.
Log files:
https://drive.google.com/open?id=1QViho0khiMQpYDOcF5ACZV2cwymgNEMS
You could find the issue reproduce step in below mail:
http://lists.starlingx.io/pipermail/starlingx-discuss/2019-June/005033.html

It seems it is pci-sriov related issue. From the sm.log, it seems Ethernet interface is not stable, and cause several services cannot run successfully, and lead to the shutdown.
Maybe you could provide some workaround suggestion for him.
Thanks.

Best Regards
Shuicheng

From: MacDonald, Eric [mailto:Eric.MacDonald at windriver.com]
Sent: Monday, June 24, 2019 7:18 PM
To: Liu, Tao <Tao.Liu at windriver.com<mailto:Tao.Liu at windriver.com>>; Ezpeer Chen <ezpeerchen at gmail.com<mailto:ezpeerchen at gmail.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] How to turn off fault management?

Hi Ezpeer,

In addition to Tao’s point …

The only time maintenance will power off a host outside of explicit administrative action is if that host’s board management controller is provisioned, the critical action for a sensor group has been changed to power cycle AND a sensor in that sensor group reports a debounced critical severity.

If you are experiencing heartbeat failures that you are trying to debug you can change the heartbeat failure action to ‘degrade’ or ‘alarm’ only to avoid the recovery reboot. Not recommended, but available for debug.

>  system service-parameter-modify platform maintenance heartbeat_failure_action=degrade

>  system service-parameter-apply platform

Locking a host will prevent host watchdog reboot due to quorum process failure or watchdog pet failure/timeout

If you are experiencing autonomous host power off then I would look at the BMC logs for critical or fatal event reports.

Eric.

From: Liu, Tao [mailto:Tao.Liu at windriver.com]
Sent: Friday, June 21, 2019 9:33 AM
To: Ezpeer Chen; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] How to turn off fault management?

Hi Ezpeer,

The fault management reports fault conditions and significant events in the system and it does not reboot or power off the controller. The maintenance system takes proper actions to recover the system When necessary.

I suggest you to view the active alarms and event history to see what failures might lead to reboot the controller for recovery. (could it be a configuration failure?).
fm alarm-list
fm event-list

In addition, /var/log/mtcAgent.log provides more details on why the host is reboot or power-off.

Regards,
Tao

From: Ezpeer Chen [mailto:ezpeerchen at gmail.com]
Sent: Friday, June 21, 2019 4:08 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] How to turn off fault management?

Dear all,

Environment: STX 1.0 (2018/10) all-in-one simplex


How could i turn off fault management which cause my system(controller-0) reboot or power-off?



Best Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190624/a0e4fd63/attachment-0001.html>