[Starlingx-discuss] How to turn off fault management?

[Ezpeer Chen]

Dear all,

Will the fault management to do actions (reboot or shutdown) based on BMC's
sensor status ?


Thanks

Qian, Bin <Bin.Qian at windriver.com> 於 2019年6月25日 週二 上午2:23寫道：

> SM only requires mtce to reboot a failed controller during a failover,
> i.e, in a duplex environment, the survivor controller requires mtce to
> reboot the failed controller.
> SM won't require reboot when there is only 1 controller available (e.g an
> aio simplex).
>
> Bin
> ------------------------------
> *From:* MacDonald, Eric [Eric.MacDonald at windriver.com]
> *Sent:* Monday, June 24, 2019 5:15 AM
> *To:* Lin, Shuicheng; Liu, Tao; Ezpeer Chen;
> starlingx-discuss at lists.starlingx.io
> *Subject:* Re: [Starlingx-discuss] How to turn off fault management?
>
> Yes, SM can request mtce to reboot a controller and there will be an
> explicit mtce log for that (below)
>
>
>
> controller-? is being force failed by SM
>
>
>
> *From:* Lin, Shuicheng [mailto:shuicheng.lin at intel.com]
> *Sent:* Monday, June 24, 2019 7:48 AM
> *To:* MacDonald, Eric; Liu, Tao; Ezpeer Chen;
> starlingx-discuss at lists.starlingx.io
> *Subject:* RE: [Starlingx-discuss] How to turn off fault management?
> *Importance:* High
>
>
>
> Hi Eric,
>
> I guess you mean sm should reboot a host, not shutdown. Is it correct?
>
> But per Ezpeer’s description, it seems like shutdown.
>
> BTW, it is with STX 1.0, from 2010.10 release branch.
>
>
>
> Best Regards
>
> Shuicheng
>
>
>
> *From:* MacDonald, Eric [mailto:Eric.MacDonald at windriver.com]
> *Sent:* Monday, June 24, 2019 7:40 PM
> *To:* Lin, Shuicheng <shuicheng.lin at intel.com>; Liu, Tao <
> Tao.Liu at windriver.com>; Ezpeer Chen <ezpeerchen at gmail.com>;
> starlingx-discuss at lists.starlingx.io
> *Subject:* RE: [Starlingx-discuss] How to turn off fault management?
>
>
>
> SM does not power off a host.
>
> When you say shutdown you mean power off correct ?
>
>
>
> *From:* Lin, Shuicheng [mailto:shuicheng.lin at intel.com
> <shuicheng.lin at intel.com>]
> *Sent:* Monday, June 24, 2019 7:38 AM
> *To:* MacDonald, Eric; Liu, Tao; Ezpeer Chen;
> starlingx-discuss at lists.starlingx.io
> *Subject:* RE: [Starlingx-discuss] How to turn off fault management?
> *Importance:* High
>
>
>
> Hi Eric,
>
> Here is the collect log for the issue.
>
> Log files:
> https://drive.google.com/open?id=1QViho0khiMQpYDOcF5ACZV2cwymgNEMS
>
> You could find the issue reproduce step in below mail:
>
> http://lists.starlingx.io/pipermail/starlingx-discuss/2019-June/005033.html
>
>
>
> It seems it is pci-sriov related issue. From the sm.log, it seems Ethernet
> interface is not stable, and cause several services cannot run
> successfully, and lead to the shutdown.
>
> Maybe you could provide some workaround suggestion for him.
>
> Thanks.
>
>
>
> Best Regards
>
> Shuicheng
>
>
>
> *From:* MacDonald, Eric [mailto:Eric.MacDonald at windriver.com
> <Eric.MacDonald at windriver.com>]
> *Sent:* Monday, June 24, 2019 7:18 PM
> *To:* Liu, Tao <Tao.Liu at windriver.com>; Ezpeer Chen <ezpeerchen at gmail.com>;
> starlingx-discuss at lists.starlingx.io
> *Subject:* Re: [Starlingx-discuss] How to turn off fault management?
>
>
>
> Hi Ezpeer,
>
>
>
> In addition to Tao’s point …
>
>
>
> The only time maintenance will power off a host outside of explicit
> administrative action is if that host’s board management controller is
> provisioned, the critical action for a sensor group has been changed to
> power cycle AND a sensor in that sensor group reports a debounced critical
> severity.
>
>
>
> If you are experiencing heartbeat failures that you are trying to debug
> you can change the heartbeat failure action to ‘degrade’ or ‘alarm’ only to
> avoid the recovery reboot. Not recommended, but available for debug.
>
> Ø  system service-parameter-modify platform maintenance
> heartbeat_failure_action=degrade
>
> Ø  system service-parameter-apply platform
>
>
>
> Locking a host will prevent host watchdog reboot due to quorum process
> failure or watchdog pet failure/timeout
>
>
>
> If you are experiencing autonomous host power off then I would look at the
> BMC logs for critical or fatal event reports.
>
>
>
> Eric.
>
>
>
> *From:* Liu, Tao [mailto:Tao.Liu at windriver.com <Tao.Liu at windriver.com>]
> *Sent:* Friday, June 21, 2019 9:33 AM
> *To:* Ezpeer Chen; starlingx-discuss at lists.starlingx.io
> *Subject:* Re: [Starlingx-discuss] How to turn off fault management?
>
>
>
> Hi Ezpeer,
>
>
>
> The fault management reports fault conditions and significant events in
> the system and it does not reboot or power off the controller. The
> maintenance system takes proper actions to recover the system When
> necessary.
>
>
>
> I suggest you to view the active alarms and event history to see what
> failures might lead to reboot the controller for recovery. (could it be a
> configuration failure?).
>
> fm alarm-list
>
> fm event-list
>
>
>
> In addition, /var/log/mtcAgent.log provides more details on why the host
> is reboot or power-off.
>
>
>
> Regards,
>
> Tao
>
>
>
> *From:* Ezpeer Chen [mailto:ezpeerchen at gmail.com <ezpeerchen at gmail.com>]
> *Sent:* Friday, June 21, 2019 4:08 AM
> *To:* starlingx-discuss at lists.starlingx.io
> *Subject:* [Starlingx-discuss] How to turn off fault management?
>
>
>
> Dear all,
>
> Environment: STX 1.0 (2018/10) all-in-one simplex
>
>
> How could i turn off fault management which cause my system(controller-0)
> reboot or power-off?
>
>
>
> Best Regards
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190626/82c20eec/attachment-0001.html>