Hi STX gurus,

 

Currently I met an issue of standby controller and worker nodes keep rebooting due to the lost of heartbeat in STX3, hope someone could give me some help. Thanks very much!

 

My system contains the following configuration:

 

Controllers: Two Inspur nf8260m5 servers each with 2 Intel X550T NICs, 1 X550 NIC for OAM network and the other for mgmt network.

Worker node: one x86_64 server with 2 Intel X722 NICs, 1 X722 NIC for mgmt. network and the other for data.

The 2 OAM network NICs are all connected to a switch, and the 3 mgmt NICs are connected to another switch.

 

I follow the STX installation guide of the deployment option Standard with Controller Storage, but now I’m facing an issue of the controller-0(stand-by controller) and worker-0(the only work node) keeps rebooting.

 

Though I can successfully ping to controller-0 and worker-0 from my active controller-1, and vice versa. I still see the following log in /var/log/mtcAgent.log.  

 

------ log excerpt from mtcAgent.log -----

2020-03-20T07:42:02.837 [2285790.01035] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)

2020-03-20T07:42:02.837 [2285790.01036] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 1 enabled hosts (threshold:2) (1:Mgmnt:10) (0:Clstr:0)

2020-03-20T07:42:03.047 [2285790.01037] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)

2020-03-20T07:42:03.047 [2285790.01038] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***

2020-03-20T07:42:03.047 [2285790.01039] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 degraded (seq:97)

2020-03-20T07:42:03.468 [2285790.01040] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)

2020-03-20T07:42:03.468 [2285790.01041] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***

2020-03-20T07:42:03.468 [2285790.01042] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure

2020-03-20T07:42:03.468 [2285790.01043] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 failed (seq:98)

2020-03-20T07:42:03.468 [2285790.01044] controller-1 mtcAgent hbs nodeClass.cpp     (4965) manage_heartbeat_failure: Warn : controller-0 starting graceful recovery

2020-03-20T07:42:03.468 [2285790.01045] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 473)

2020-03-20T07:42:03.468 [2285790.01046] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent

2020-03-20T07:42:03.468 [2285790.01047] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent

2020-03-20T07:42:03.468 [2285790.01048] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1684) recovery_handler        : Warn : controller-0 Graceful Recovery (1 of 3)

2020-03-20T07:42:03.468 [2285790.01049] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Graceful Recovery (seq:99)

2020-03-20T07:42:03.468 [2285790.01050] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:03.468 [2285790.01051] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)

2020-03-20T07:42:03.468 [2285790.01052] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:05.473 [2285790.01053] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-enabled-failed) (stage:0)

2020-03-20T07:42:05.473 [2285790.01054] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1729) recovery_handler        : Info : controller-0 requesting mtcAlive with 5 sec timeout

2020-03-20T07:42:05.473 [2285790.01055] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (enabled-failed)

2020-03-20T07:42:05.473 [2285790.01056] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1747) recovery_handler        : Info : controller-0 got requested mtcAlive

2020-03-20T07:42:05.473 [2285790.01057] controller-1 mtcAgent |-| nodeClass.cpp     (2555) stop_offline_handler    : Info : controller-0 stopping offline handler (unlocked-enabled-failed) (stage:3)

2020-03-20T07:42:05.473 [2285790.01058] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1814) recovery_handler        : Warn : controller-0 Connectivity Recovered ; host did not reset

2020-03-20T07:42:05.473 [2285790.01059] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1816) recovery_handler        : Warn : controller-0 ... continuing with graceful recovery

2020-03-20T07:42:05.473 [2285790.01060] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1817) recovery_handler        : Warn : controller-0 ... with no affect to host services

2020-03-20T07:42:05.473 [2285790.01061] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-0 hbsAgent

2020-03-20T07:42:05.473 [2285790.01062] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-1 hbsAgent

2020-03-20T07:42:05.473 [2285790.01063] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (2489) recovery_handler        : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event)

2020-03-20T07:42:05.474 [2285790.01064] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:05.474 [2285790.01065] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:05.985 [2285790.01066] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)

2020-03-20T07:42:05.985 [2285790.01067] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 1 enabled hosts (threshold:2) (1:Mgmnt:11) (0:Clstr:0)

2020-03-20T07:42:06.196 [2285790.01068] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)

2020-03-20T07:42:06.196 [2285790.01069] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***

2020-03-20T07:42:06.616 [2285790.01070] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)

2020-03-20T07:42:06.616 [2285790.01071] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***

2020-03-20T07:42:06.616 [2285790.01072] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure

2020-03-20T07:42:06.616 [2285790.01073] controller-1 mtcAgent hbs nodeClass.cpp     (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery

2020-03-20T07:42:06.616 [2285790.01074] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 478)

2020-03-20T07:42:06.616 [2285790.01075] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent

2020-03-20T07:42:06.616 [2285790.01076] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent

2020-03-20T07:42:06.616 [2285790.01077] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1684) recovery_handler        : Warn : controller-0 Graceful Recovery (2 of 3)

2020-03-20T07:42:06.616 [2285790.01078] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Graceful Recovery Retry (seq:100)

2020-03-20T07:42:06.616 [2285790.01079] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:06.616 [2285790.01080] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)

2020-03-20T07:42:06.616 [2285790.01081] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:08.569 [2285790.01082] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1421) enable_handler          : Info : worker-0 heartbeating

2020-03-20T07:42:08.574 [2285790.01083] controller-1 mtcAgent inv mtcInvApi.cpp     ( 437) mtcInvApi_force_task    : Info : worker-0 task clear (seq:96) (was:Enabling)

2020-03-20T07:42:08.574 fmAPI.cpp(490): Enqueue raise alarm request: UUID (b24f0d21-6c0d-4862-99de-6e8ad1f04b99) alarm id (200.022) instant id (host=worker-0.state=enabled)

2020-03-20T07:42:08.574 [2285790.01084] controller-1 mtcAgent hbs nodeClass.cpp     (6188) allStateChange          : Info : worker-0 unlocked-enabled-degraded (seq:97)

2020-03-20T07:42:08.605 fmAlarmUtils.cpp(624): Sending FM raise alarm request: alarm_id (200.022), entity_id (host=worker-0.state=enabled)

2020-03-20T07:42:08.620 [2285790.01085] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-enabled-failed) (stage:0)

2020-03-20T07:42:08.620 [2285790.01086] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1729) recovery_handler        : Info : controller-0 requesting mtcAlive with 5 sec timeout

2020-03-20T07:42:08.620 [2285790.01087] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (enabled-failed)

2020-03-20T07:42:08.620 [2285790.01088] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1747) recovery_handler        : Info : controller-0 got requested mtcAlive

2020-03-20T07:42:08.620 [2285790.01089] controller-1 mtcAgent |-| nodeClass.cpp     (2555) stop_offline_handler    : Info : controller-0 stopping offline handler (unlocked-enabled-failed) (stage:3)

2020-03-20T07:42:08.620 [2285790.01090] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1814) recovery_handler        : Warn : controller-0 Connectivity Recovered ; host did not reset

2020-03-20T07:42:08.620 [2285790.01091] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1816) recovery_handler        : Warn : controller-0 ... continuing with graceful recovery

2020-03-20T07:42:08.620 [2285790.01092] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1817) recovery_handler        : Warn : controller-0 ... with no affect to host services

2020-03-20T07:42:08.621 [2285790.01093] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-0 hbsAgent

2020-03-20T07:42:08.621 [2285790.01094] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-1 hbsAgent

2020-03-20T07:42:08.621 [2285790.01095] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (2489) recovery_handler        : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event)

2020-03-20T07:42:08.621 [2285790.01096] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:09.134 [2285790.01104] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)

2020-03-20T07:42:09.134 [2285790.01105] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 2 enabled hosts (threshold:2) (1:Mgmnt:12) (0:Clstr:0)

2020-03-20T07:42:09.344 [2285790.01106] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)

2020-03-20T07:42:09.344 [2285790.01107] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***

2020-03-20T07:42:09.764 [2285790.01108] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)

2020-03-20T07:42:09.764 [2285790.01109] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***

2020-03-20T07:42:09.764 [2285790.01110] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure

2020-03-20T07:42:09.764 [2285790.01111] controller-1 mtcAgent hbs nodeClass.cpp     (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery

2020-03-20T07:42:09.765 [2285790.01112] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 481)

2020-03-20T07:42:09.765 [2285790.01113] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent

2020-03-20T07:42:09.765 [2285790.01114] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent

2020-03-20T07:42:09.765 [2285790.01115] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1684) recovery_handler        : Warn : controller-0 Graceful Recovery (3 of 3)

2020-03-20T07:42:09.765 [2285790.01116] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Graceful Recovery Retry (seq:101)

2020-03-20T07:42:09.765 [2285790.01117] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:09.765 [2285790.01118] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)

2020-03-20T07:42:09.765 [2285790.01119] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:10.468 [2285790.01120] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3473) online_handler          : Info : controller-0 mtcAlive ; going 'online'

2020-03-20T07:42:10.468 [2285790.01121] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 online (seq:102)

2020-03-20T07:42:11.770 [2285790.01122] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-enabled-online) (stage:0)

2020-03-20T07:42:11.770 [2285790.01123] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1729) recovery_handler        : Info : controller-0 requesting mtcAlive with 5 sec timeout

2020-03-20T07:42:11.770 [2285790.01124] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (enabled-online)

2020-03-20T07:42:11.770 [2285790.01125] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1747) recovery_handler        : Info : controller-0 got requested mtcAlive

2020-03-20T07:42:11.770 [2285790.01126] controller-1 mtcAgent |-| nodeClass.cpp     (2555) stop_offline_handler    : Info : controller-0 stopping offline handler (unlocked-enabled-online) (stage:3)

2020-03-20T07:42:11.770 [2285790.01127] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1814) recovery_handler        : Warn : controller-0 Connectivity Recovered ; host did not reset

2020-03-20T07:42:11.770 [2285790.01128] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1816) recovery_handler        : Warn : controller-0 ... continuing with graceful recovery

2020-03-20T07:42:11.770 [2285790.01129] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1817) recovery_handler        : Warn : controller-0 ... with no affect to host services

2020-03-20T07:42:11.770 [2285790.01130] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-0 hbsAgent

2020-03-20T07:42:11.770 [2285790.01131] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-1 hbsAgent

2020-03-20T07:42:11.770 [2285790.01132] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (2489) recovery_handler        : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event)

2020-03-20T07:42:11.770 [2285790.01133] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:11.770 [2285790.01134] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:12.282 [2285790.01135] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)

2020-03-20T07:42:12.282 [2285790.01136] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 2 enabled hosts (threshold:2) (1:Mgmnt:13) (0:Clstr:0)

2020-03-20T07:42:12.493 [2285790.01137] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)

2020-03-20T07:42:12.493 [2285790.01138] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***

2020-03-20T07:42:12.913 [2285790.01139] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)

2020-03-20T07:42:12.913 [2285790.01140] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***

2020-03-20T07:42:12.913 [2285790.01141] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure

2020-03-20T07:42:12.913 [2285790.01142] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 failed (seq:103)

2020-03-20T07:42:12.914 [2285790.01143] controller-1 mtcAgent hbs nodeClass.cpp     (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery

2020-03-20T07:42:12.914 [2285790.01144] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 484)

2020-03-20T07:42:12.914 [2285790.01145] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent

2020-03-20T07:42:12.914 [2285790.01146] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent

2020-03-20T07:42:12.914 [2285790.01147] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1671) recovery_handler        :Error : controller-0 Graceful Recovery Failed (retries=4)

2020-03-20T07:42:12.914 [2285790.01148] controller-1 mtcAgent |-| nodeClass.cpp     (7590) force_full_enable       : Info : controller-0 Forcing Full Enable Sequence

2020-03-20T07:42:12.914 [2285790.01149] controller-1 mtcAgent hbs nodeClass.cpp     (1681) alarm_enabled_failure   :Error : controller-0 critical enable failure

2020-03-20T07:42:12.914 [2285790.01150] controller-1 mtcAgent alm mtcAlarm.cpp      ( 417) mtcAlarm_critical       :Error : controller-0 setting critical 'In-Service' failure alarm (200.004 )

2020-03-20T07:42:12.914 fmAPI.cpp(490): Enqueue raise alarm request: UUID (4f1c52ab-098d-4b87-aafc-acfdd2c7ac7a) alarm id (200.004) instant id (host=controller-0)

2020-03-20T07:42:12.914 fmAPI.cpp(490): Enqueue raise alarm request: UUID (d6a99af3-2e3b-4bf4-b0db-07f966fafea2) alarm id (200.022) instant id (host=controller-0.state=disabled)

2020-03-20T07:42:12.914 [2285790.01151] controller-1 mtcAgent hbs nodeClass.cpp     (6188) allStateChange          : Info : controller-0 unlocked-disabled-failed (seq:104)

2020-03-20T07:42:12.914 [2285790.01152] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:12.914 [2285790.01153] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)

2020-03-20T07:42:12.914 [2285790.01154] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  ( 573) enable_handler          :Error : controller-0 Main Enable FSM (from failed)

2020-03-20T07:42:12.914 [2285790.01155] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent

2020-03-20T07:42:12.914 [2285790.01156] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent

2020-03-20T07:42:12.914 [2285790.01157] controller-1 mtcAgent vim mtcVimApi.cpp     ( 251) mtcVimApi_state_change  :Error : controller-0 {"state-change": {"administrative":"unlocked","operational":"disabled","availability":"failed","subfunction_oper":"disabled","subfunction_avail":"not-installed"},"hostname":"controller-0","uuid":"28412a2d-fb95-472d-9797-c2ef4b18752f","subfunctions":"controller","personality":"controller"}

2020-03-20T07:42:12.914 [2285790.01158] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:12.914 [2285790.01159] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:12.914 [2285790.01160] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  ( 769) enable_handler          : Info : controller-0 Main Enable FSM (from start)

2020-03-20T07:42:12.914 [2285790.01161] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 750) workQueue_purge         : Warn : controller-0 purging 4 items from workQueue

2020-03-20T07:42:12.914 [2285790.01162] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 764) workQueue_purge         : Warn : controller-0 mtcInvApi_update_state seq:103 ... did not complete (Wait)

2020-03-20T07:42:12.914 [2285790.01163] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 758) workQueue_purge         : Warn : controller-0 mtcInvApi_force_states seq:104 ... was not executed

2020-03-20T07:42:12.914 [2285790.01164] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 758) workQueue_purge         : Warn : controller-0 mtcInvApi_force_states seq:105 ... was not executed

2020-03-20T07:42:12.914 [2285790.01165] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 758) workQueue_purge         : Warn : controller-0 mtcVimApi_state_change seq:106 ... was not executed

2020-03-20T07:42:12.914 [2285790.01166] controller-1 mtcAgent mgr mtcSmgrApi.cpp    ( 209) mtcSmgrApi_request      : Info : controller-0 sending 'unlocked-disabled' request to HA Service Manager

2020-03-20T07:42:12.919 [2285790.01167] controller-1 mtcAgent mgr mtcSmgrApi.cpp    ( 300) mtcSmgrApi_request      : Info : controller-0 is 'disabled' to Service Management

2020-03-20T07:42:12.919 [2285790.01168] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:12.919 [2285790.01169] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent

2020-03-20T07:42:12.919 [2285790.01170] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent

2020-03-20T07:42:12.919 [2285790.01171] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)

2020-03-20T07:42:12.919 [2285790.01172] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  ( 963) enable_handler          : Info : controller-0 reboot

2020-03-20T07:42:12.919 [2285790.01173] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (  91) calc_reset_prog_timeout : Info : controller-0 Reboot/Reset progression has 40 sec 'wait for offline' timeout

2020-03-20T07:42:12.919 [2285790.01174] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (  95) calc_reset_prog_timeout : Info : controller-0 ... sources - mgmnt:Yes  clstr:No  bmc:No

2020-03-20T07:42:12.919 [2285790.01175] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)

2020-03-20T07:42:12.924 [2285790.01176] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Reboot Request (seq:108)

2020-03-20T07:42:12.924 [2285790.01177] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-disabled-failed) (stage:0)

2020-03-20T07:42:12.924 [2285790.01178] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (disabled-failed)

2020-03-20T07:42:12.924 [2285790.01179] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 629) send_mtc_cmd            : Info : controller-0 sending 'reboot' request (Mgmnt network)

2020-03-20T07:42:12.924 [2285790.01180] controller-1 mtcAgent cmd mtcCmdHdlr.cpp    ( 403) cmd_handler             : Info : controller-0 waiting for REBOOT ACK

2020-03-20T07:42:12.924 [2285790.01181] controller-1 mtcAgent |-| mtcCmdHdlr.cpp    ( 437) cmd_handler             : Info : controller-0 REBOOT Request Succeeded

2020-03-20T07:42:12.924 [2285790.01182] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Rebooting (seq:109)

2020-03-20T07:42:12.924 [2285790.01183] controller-1 mtcAgent cmd mtcCmdHdlr.cpp    ( 459) cmd_handler             : Info : controller-0 searching for offline ; next reboot attempt in 7 seconds

 

BR,

-Lianhao