[Starlingx-discuss] How to turn off fault management?

Wed Jun 26 11:53:37 UTC 2019

As mentioned below
“The only time maintenance will power off a host outside of explicit administrative action is if that host’s board management controller is provisioned, the critical action for a sensor group has been changed to power cycle AND a sensor in that sensor group reports a debounced critical severity.”
You would see customer logs to this effect.
Eric.

From: Ezpeer Chen [mailto:ezpeerchen at gmail.com]
Sent: Wednesday, June 26, 2019 3:56 AM
To: Qian, Bin
Cc: MacDonald, Eric; Lin, Shuicheng; Liu, Tao; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] How to turn off fault management?

Dear all,

Will the fault management to do actions (reboot or shutdown) based on BMC's sensor status ?

Thanks

Qian, Bin <Bin.Qian at windriver.com<mailto:Bin.Qian at windriver.com>> 於 2019年6月25日 週二 上午2:23寫道：
SM only requires mtce to reboot a failed controller during a failover, i.e, in a duplex environment, the survivor controller requires mtce to reboot the failed controller.
SM won't require reboot when there is only 1 controller available (e.g an aio simplex).

Bin
________________________________
From: MacDonald, Eric [Eric.MacDonald at windriver.com<mailto:Eric.MacDonald at windriver.com>]
Sent: Monday, June 24, 2019 5:15 AM
To: Lin, Shuicheng; Liu, Tao; Ezpeer Chen; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] How to turn off fault management?
Yes, SM can request mtce to reboot a controller and there will be an explicit mtce log for that (below)

controller-? is being force failed by SM

From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>]
Sent: Monday, June 24, 2019 7:48 AM
To: MacDonald, Eric; Liu, Tao; Ezpeer Chen; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] How to turn off fault management?
Importance: High

Hi Eric,
I guess you mean sm should reboot a host, not shutdown. Is it correct?
But per Ezpeer’s description, it seems like shutdown.
BTW, it is with STX 1.0, from 2010.10 release branch.

Best Regards
Shuicheng

From: MacDonald, Eric [mailto:Eric.MacDonald at windriver.com<mailto:Eric.MacDonald at windriver.com>]
Sent: Monday, June 24, 2019 7:40 PM
To: Lin, Shuicheng <shuicheng.lin at intel.com<mailto:shuicheng.lin at intel.com>>; Liu, Tao <Tao.Liu at windriver.com<mailto:Tao.Liu at windriver.com>>; Ezpeer Chen <ezpeerchen at gmail.com<mailto:ezpeerchen at gmail.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] How to turn off fault management?

SM does not power off a host.
When you say shutdown you mean power off correct ?

From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
Sent: Monday, June 24, 2019 7:38 AM
To: MacDonald, Eric; Liu, Tao; Ezpeer Chen; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] How to turn off fault management?
Importance: High

Hi Eric,
Here is the collect log for the issue.
Log files:
https://drive.google.com/open?id=1QViho0khiMQpYDOcF5ACZV2cwymgNEMS
You could find the issue reproduce step in below mail:
http://lists.starlingx.io/pipermail/starlingx-discuss/2019-June/005033.html

It seems it is pci-sriov related issue. From the sm.log, it seems Ethernet interface is not stable, and cause several services cannot run successfully, and lead to the shutdown.
Maybe you could provide some workaround suggestion for him.
Thanks.

Best Regards
Shuicheng

From: MacDonald, Eric [mailto:Eric.MacDonald at windriver.com]
Sent: Monday, June 24, 2019 7:18 PM
To: Liu, Tao <Tao.Liu at windriver.com<mailto:Tao.Liu at windriver.com>>; Ezpeer Chen <ezpeerchen at gmail.com<mailto:ezpeerchen at gmail.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] How to turn off fault management?

Hi Ezpeer,

In addition to Tao’s point …

The only time maintenance will power off a host outside of explicit administrative action is if that host’s board management controller is provisioned, the critical action for a sensor group has been changed to power cycle AND a sensor in that sensor group reports a debounced critical severity.

If you are experiencing heartbeat failures that you are trying to debug you can change the heartbeat failure action to ‘degrade’ or ‘alarm’ only to avoid the recovery reboot. Not recommended, but available for debug.

>  system service-parameter-modify platform maintenance heartbeat_failure_action=degrade

>  system service-parameter-apply platform

Locking a host will prevent host watchdog reboot due to quorum process failure or watchdog pet failure/timeout

If you are experiencing autonomous host power off then I would look at the BMC logs for critical or fatal event reports.

Eric.

From: Liu, Tao [mailto:Tao.Liu at windriver.com]
Sent: Friday, June 21, 2019 9:33 AM
To: Ezpeer Chen; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] How to turn off fault management?

Hi Ezpeer,

The fault management reports fault conditions and significant events in the system and it does not reboot or power off the controller. The maintenance system takes proper actions to recover the system When necessary.

I suggest you to view the active alarms and event history to see what failures might lead to reboot the controller for recovery. (could it be a configuration failure?).
fm alarm-list
fm event-list

In addition, /var/log/mtcAgent.log provides more details on why the host is reboot or power-off.

Regards,
Tao

From: Ezpeer Chen [mailto:ezpeerchen at gmail.com]
Sent: Friday, June 21, 2019 4:08 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] How to turn off fault management?

Dear all,

Environment: STX 1.0 (2018/10) all-in-one simplex

How could i turn off fault management which cause my system(controller-0) reboot or power-off?

Best Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190626/a2ef3c71/attachment-0001.html>