[Starlingx-discuss] STX3 heartbeat lost causing controller and worker nodes rebooting

Lu, Lianhao lianhao.lu at intel.com
Fri Mar 20 07:57:44 UTC 2020


Hi STX gurus,

Currently I met an issue of standby controller and worker nodes keep rebooting due to the lost of heartbeat in STX3, hope someone could give me some help. Thanks very much!

My system contains the following configuration:

Controllers: Two Inspur nf8260m5 servers each with 2 Intel X550T NICs, 1 X550 NIC for OAM network and the other for mgmt network.
Worker node: one x86_64 server with 2 Intel X722 NICs, 1 X722 NIC for mgmt. network and the other for data.
The 2 OAM network NICs are all connected to a switch, and the 3 mgmt NICs are connected to another switch.

I follow the STX installation guide of the deployment option Standard with Controller Storage, but now I'm facing an issue of the controller-0(stand-by controller) and worker-0(the only work node) keeps rebooting.

Though I can successfully ping to controller-0 and worker-0 from my active controller-1, and vice versa. I still see the following log in /var/log/mtcAgent.log.

------ log excerpt from mtcAgent.log -----
2020-03-20T07:42:02.837 [2285790.01035] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)
2020-03-20T07:42:02.837 [2285790.01036] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 1 enabled hosts (threshold:2) (1:Mgmnt:10) (0:Clstr:0)
2020-03-20T07:42:03.047 [2285790.01037] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)
2020-03-20T07:42:03.047 [2285790.01038] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***
2020-03-20T07:42:03.047 [2285790.01039] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 degraded (seq:97)
2020-03-20T07:42:03.468 [2285790.01040] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)
2020-03-20T07:42:03.468 [2285790.01041] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***
2020-03-20T07:42:03.468 [2285790.01042] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure
2020-03-20T07:42:03.468 [2285790.01043] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 failed (seq:98)
2020-03-20T07:42:03.468 [2285790.01044] controller-1 mtcAgent hbs nodeClass.cpp     (4965) manage_heartbeat_failure: Warn : controller-0 starting graceful recovery
2020-03-20T07:42:03.468 [2285790.01045] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 473)
2020-03-20T07:42:03.468 [2285790.01046] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent
2020-03-20T07:42:03.468 [2285790.01047] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent
2020-03-20T07:42:03.468 [2285790.01048] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1684) recovery_handler        : Warn : controller-0 Graceful Recovery (1 of 3)
2020-03-20T07:42:03.468 [2285790.01049] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Graceful Recovery (seq:99)
2020-03-20T07:42:03.468 [2285790.01050] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:03.468 [2285790.01051] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)
2020-03-20T07:42:03.468 [2285790.01052] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:05.473 [2285790.01053] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-enabled-failed) (stage:0)
2020-03-20T07:42:05.473 [2285790.01054] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1729) recovery_handler        : Info : controller-0 requesting mtcAlive with 5 sec timeout
2020-03-20T07:42:05.473 [2285790.01055] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (enabled-failed)
2020-03-20T07:42:05.473 [2285790.01056] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1747) recovery_handler        : Info : controller-0 got requested mtcAlive
2020-03-20T07:42:05.473 [2285790.01057] controller-1 mtcAgent |-| nodeClass.cpp     (2555) stop_offline_handler    : Info : controller-0 stopping offline handler (unlocked-enabled-failed) (stage:3)
2020-03-20T07:42:05.473 [2285790.01058] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1814) recovery_handler        : Warn : controller-0 Connectivity Recovered ; host did not reset
2020-03-20T07:42:05.473 [2285790.01059] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1816) recovery_handler        : Warn : controller-0 ... continuing with graceful recovery
2020-03-20T07:42:05.473 [2285790.01060] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1817) recovery_handler        : Warn : controller-0 ... with no affect to host services
2020-03-20T07:42:05.473 [2285790.01061] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-0 hbsAgent
2020-03-20T07:42:05.473 [2285790.01062] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-1 hbsAgent
2020-03-20T07:42:05.473 [2285790.01063] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (2489) recovery_handler        : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event)
2020-03-20T07:42:05.474 [2285790.01064] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:05.474 [2285790.01065] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:05.985 [2285790.01066] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)
2020-03-20T07:42:05.985 [2285790.01067] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 1 enabled hosts (threshold:2) (1:Mgmnt:11) (0:Clstr:0)
2020-03-20T07:42:06.196 [2285790.01068] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)
2020-03-20T07:42:06.196 [2285790.01069] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***
2020-03-20T07:42:06.616 [2285790.01070] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)
2020-03-20T07:42:06.616 [2285790.01071] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***
2020-03-20T07:42:06.616 [2285790.01072] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure
2020-03-20T07:42:06.616 [2285790.01073] controller-1 mtcAgent hbs nodeClass.cpp     (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery
2020-03-20T07:42:06.616 [2285790.01074] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 478)
2020-03-20T07:42:06.616 [2285790.01075] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent
2020-03-20T07:42:06.616 [2285790.01076] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent
2020-03-20T07:42:06.616 [2285790.01077] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1684) recovery_handler        : Warn : controller-0 Graceful Recovery (2 of 3)
2020-03-20T07:42:06.616 [2285790.01078] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Graceful Recovery Retry (seq:100)
2020-03-20T07:42:06.616 [2285790.01079] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:06.616 [2285790.01080] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)
2020-03-20T07:42:06.616 [2285790.01081] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:08.569 [2285790.01082] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1421) enable_handler          : Info : worker-0 heartbeating
2020-03-20T07:42:08.574 [2285790.01083] controller-1 mtcAgent inv mtcInvApi.cpp     ( 437) mtcInvApi_force_task    : Info : worker-0 task clear (seq:96) (was:Enabling)
2020-03-20T07:42:08.574 fmAPI.cpp(490): Enqueue raise alarm request: UUID (b24f0d21-6c0d-4862-99de-6e8ad1f04b99) alarm id (200.022) instant id (host=worker-0.state=enabled)
2020-03-20T07:42:08.574 [2285790.01084] controller-1 mtcAgent hbs nodeClass.cpp     (6188) allStateChange          : Info : worker-0 unlocked-enabled-degraded (seq:97)
2020-03-20T07:42:08.605 fmAlarmUtils.cpp(624): Sending FM raise alarm request: alarm_id (200.022), entity_id (host=worker-0.state=enabled)
2020-03-20T07:42:08.620 [2285790.01085] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-enabled-failed) (stage:0)
2020-03-20T07:42:08.620 [2285790.01086] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1729) recovery_handler        : Info : controller-0 requesting mtcAlive with 5 sec timeout
2020-03-20T07:42:08.620 [2285790.01087] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (enabled-failed)
2020-03-20T07:42:08.620 [2285790.01088] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1747) recovery_handler        : Info : controller-0 got requested mtcAlive
2020-03-20T07:42:08.620 [2285790.01089] controller-1 mtcAgent |-| nodeClass.cpp     (2555) stop_offline_handler    : Info : controller-0 stopping offline handler (unlocked-enabled-failed) (stage:3)
2020-03-20T07:42:08.620 [2285790.01090] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1814) recovery_handler        : Warn : controller-0 Connectivity Recovered ; host did not reset
2020-03-20T07:42:08.620 [2285790.01091] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1816) recovery_handler        : Warn : controller-0 ... continuing with graceful recovery
2020-03-20T07:42:08.620 [2285790.01092] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1817) recovery_handler        : Warn : controller-0 ... with no affect to host services
2020-03-20T07:42:08.621 [2285790.01093] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-0 hbsAgent
2020-03-20T07:42:08.621 [2285790.01094] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-1 hbsAgent
2020-03-20T07:42:08.621 [2285790.01095] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (2489) recovery_handler        : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event)
2020-03-20T07:42:08.621 [2285790.01096] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:09.134 [2285790.01104] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)
2020-03-20T07:42:09.134 [2285790.01105] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 2 enabled hosts (threshold:2) (1:Mgmnt:12) (0:Clstr:0)
2020-03-20T07:42:09.344 [2285790.01106] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)
2020-03-20T07:42:09.344 [2285790.01107] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***
2020-03-20T07:42:09.764 [2285790.01108] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)
2020-03-20T07:42:09.764 [2285790.01109] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***
2020-03-20T07:42:09.764 [2285790.01110] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure
2020-03-20T07:42:09.764 [2285790.01111] controller-1 mtcAgent hbs nodeClass.cpp     (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery
2020-03-20T07:42:09.765 [2285790.01112] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 481)
2020-03-20T07:42:09.765 [2285790.01113] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent
2020-03-20T07:42:09.765 [2285790.01114] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent
2020-03-20T07:42:09.765 [2285790.01115] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1684) recovery_handler        : Warn : controller-0 Graceful Recovery (3 of 3)
2020-03-20T07:42:09.765 [2285790.01116] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Graceful Recovery Retry (seq:101)
2020-03-20T07:42:09.765 [2285790.01117] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:09.765 [2285790.01118] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)
2020-03-20T07:42:09.765 [2285790.01119] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:10.468 [2285790.01120] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3473) online_handler          : Info : controller-0 mtcAlive ; going 'online'
2020-03-20T07:42:10.468 [2285790.01121] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 online (seq:102)
2020-03-20T07:42:11.770 [2285790.01122] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-enabled-online) (stage:0)
2020-03-20T07:42:11.770 [2285790.01123] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1729) recovery_handler        : Info : controller-0 requesting mtcAlive with 5 sec timeout
2020-03-20T07:42:11.770 [2285790.01124] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (enabled-online)
2020-03-20T07:42:11.770 [2285790.01125] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1747) recovery_handler        : Info : controller-0 got requested mtcAlive
2020-03-20T07:42:11.770 [2285790.01126] controller-1 mtcAgent |-| nodeClass.cpp     (2555) stop_offline_handler    : Info : controller-0 stopping offline handler (unlocked-enabled-online) (stage:3)
2020-03-20T07:42:11.770 [2285790.01127] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1814) recovery_handler        : Warn : controller-0 Connectivity Recovered ; host did not reset
2020-03-20T07:42:11.770 [2285790.01128] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1816) recovery_handler        : Warn : controller-0 ... continuing with graceful recovery
2020-03-20T07:42:11.770 [2285790.01129] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1817) recovery_handler        : Warn : controller-0 ... with no affect to host services
2020-03-20T07:42:11.770 [2285790.01130] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-0 hbsAgent
2020-03-20T07:42:11.770 [2285790.01131] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 start host service sent to controller-1 hbsAgent
2020-03-20T07:42:11.770 [2285790.01132] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (2489) recovery_handler        : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event)
2020-03-20T07:42:11.770 [2285790.01133] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:11.770 [2285790.01134] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:12.282 [2285790.01135] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor set (Mgmnt)
2020-03-20T07:42:12.282 [2285790.01136] controller-1 mtcAgent --- mtcNodeMnfa.cpp   ( 131) mnfa_add_host           : Info : controller-0 MNFA new candidate (Mgmnt) 2 enabled hosts (threshold:2) (1:Mgmnt:13) (0:Clstr:0)
2020-03-20T07:42:12.493 [2285790.01137] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade set (Mgmnt)
2020-03-20T07:42:12.493 [2285790.01138] controller-1 mtcAgent hbs nodeClass.cpp     (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss ***
2020-03-20T07:42:12.913 [2285790.01139] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat loss (Mgmnt)
2020-03-20T07:42:12.913 [2285790.01140] controller-1 mtcAgent hbs nodeClass.cpp     (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss ***
2020-03-20T07:42:12.913 [2285790.01141] controller-1 mtcAgent hbs nodeClass.cpp     (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure
2020-03-20T07:42:12.913 [2285790.01142] controller-1 mtcAgent inv mtcInvApi.cpp     (1119) mtcInvApi_update_state  : Info : controller-0 failed (seq:103)
2020-03-20T07:42:12.914 [2285790.01143] controller-1 mtcAgent hbs nodeClass.cpp     (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery
2020-03-20T07:42:12.914 [2285790.01144] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  (1643) recovery_handler        : Info : controller-0 Graceful Recovery (uptime was 484)
2020-03-20T07:42:12.914 [2285790.01145] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent
2020-03-20T07:42:12.914 [2285790.01146] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent
2020-03-20T07:42:12.914 [2285790.01147] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (1671) recovery_handler        :Error : controller-0 Graceful Recovery Failed (retries=4)
2020-03-20T07:42:12.914 [2285790.01148] controller-1 mtcAgent |-| nodeClass.cpp     (7590) force_full_enable       : Info : controller-0 Forcing Full Enable Sequence
2020-03-20T07:42:12.914 [2285790.01149] controller-1 mtcAgent hbs nodeClass.cpp     (1681) alarm_enabled_failure   :Error : controller-0 critical enable failure
2020-03-20T07:42:12.914 [2285790.01150] controller-1 mtcAgent alm mtcAlarm.cpp      ( 417) mtcAlarm_critical       :Error : controller-0 setting critical 'In-Service' failure alarm (200.004 )
2020-03-20T07:42:12.914 fmAPI.cpp(490): Enqueue raise alarm request: UUID (4f1c52ab-098d-4b87-aafc-acfdd2c7ac7a) alarm id (200.004) instant id (host=controller-0)
2020-03-20T07:42:12.914 fmAPI.cpp(490): Enqueue raise alarm request: UUID (d6a99af3-2e3b-4bf4-b0db-07f966fafea2) alarm id (200.022) instant id (host=controller-0.state=disabled)
2020-03-20T07:42:12.914 [2285790.01151] controller-1 mtcAgent hbs nodeClass.cpp     (6188) allStateChange          : Info : controller-0 unlocked-disabled-failed (seq:104)
2020-03-20T07:42:12.914 [2285790.01152] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:12.914 [2285790.01153] controller-1 mtcAgent hbs nodeClass.cpp     (4833) hbs_minor_clear         : Warn : controller-0 MNFA host tally (Mgmnt:0)
2020-03-20T07:42:12.914 [2285790.01154] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  ( 573) enable_handler          :Error : controller-0 Main Enable FSM (from failed)
2020-03-20T07:42:12.914 [2285790.01155] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent
2020-03-20T07:42:12.914 [2285790.01156] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent
2020-03-20T07:42:12.914 [2285790.01157] controller-1 mtcAgent vim mtcVimApi.cpp     ( 251) mtcVimApi_state_change  :Error : controller-0 {"state-change": {"administrative":"unlocked","operational":"disabled","availability":"failed","subfunction_oper":"disabled","subfunction_avail":"not-installed"},"hostname":"controller-0","uuid":"28412a2d-fb95-472d-9797-c2ef4b18752f","subfunctions":"controller","personality":"controller"}
2020-03-20T07:42:12.914 [2285790.01158] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:12.914 [2285790.01159] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:12.914 [2285790.01160] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  ( 769) enable_handler          : Info : controller-0 Main Enable FSM (from start)
2020-03-20T07:42:12.914 [2285790.01161] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 750) workQueue_purge         : Warn : controller-0 purging 4 items from workQueue
2020-03-20T07:42:12.914 [2285790.01162] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 764) workQueue_purge         : Warn : controller-0 mtcInvApi_update_state seq:103 ... did not complete (Wait)
2020-03-20T07:42:12.914 [2285790.01163] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 758) workQueue_purge         : Warn : controller-0 mtcInvApi_force_states seq:104 ... was not executed
2020-03-20T07:42:12.914 [2285790.01164] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 758) workQueue_purge         : Warn : controller-0 mtcInvApi_force_states seq:105 ... was not executed
2020-03-20T07:42:12.914 [2285790.01165] controller-1 mtcAgent --- mtcWorkQueue.cpp  ( 758) workQueue_purge         : Warn : controller-0 mtcVimApi_state_change seq:106 ... was not executed
2020-03-20T07:42:12.914 [2285790.01166] controller-1 mtcAgent mgr mtcSmgrApi.cpp    ( 209) mtcSmgrApi_request      : Info : controller-0 sending 'unlocked-disabled' request to HA Service Manager
2020-03-20T07:42:12.919 [2285790.01167] controller-1 mtcAgent mgr mtcSmgrApi.cpp    ( 300) mtcSmgrApi_request      : Info : controller-0 is 'disabled' to Service Management
2020-03-20T07:42:12.919 [2285790.01168] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:12.919 [2285790.01169] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-0 hbsAgent
2020-03-20T07:42:12.919 [2285790.01170] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 878) send_hbs_command        : Info : controller-0 stop host service sent to controller-1 hbsAgent
2020-03-20T07:42:12.919 [2285790.01171] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat minor clear (Mgmnt)
2020-03-20T07:42:12.919 [2285790.01172] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp  ( 963) enable_handler          : Info : controller-0 reboot
2020-03-20T07:42:12.919 [2285790.01173] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (  91) calc_reset_prog_timeout : Info : controller-0 Reboot/Reset progression has 40 sec 'wait for offline' timeout
2020-03-20T07:42:12.919 [2285790.01174] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (  95) calc_reset_prog_timeout : Info : controller-0 ... sources - mgmnt:Yes  clstr:No  bmc:No
2020-03-20T07:42:12.919 [2285790.01175] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 971) service_events          : Info : controller-0 heartbeat degrade clear (Mgmnt)
2020-03-20T07:42:12.924 [2285790.01176] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Reboot Request (seq:108)
2020-03-20T07:42:12.924 [2285790.01177] controller-1 mtcAgent |-| nodeClass.cpp     (2542) start_offline_handler   : Info : controller-0 starting offline handler (unlocked-disabled-failed) (stage:0)
2020-03-20T07:42:12.924 [2285790.01178] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp  (3242) offline_handler         : Info : controller-0 starting 100 msec offline audit (disabled-failed)
2020-03-20T07:42:12.924 [2285790.01179] controller-1 mtcAgent msg mtcCtrlMsg.cpp    ( 629) send_mtc_cmd            : Info : controller-0 sending 'reboot' request (Mgmnt network)
2020-03-20T07:42:12.924 [2285790.01180] controller-1 mtcAgent cmd mtcCmdHdlr.cpp    ( 403) cmd_handler             : Info : controller-0 waiting for REBOOT ACK
2020-03-20T07:42:12.924 [2285790.01181] controller-1 mtcAgent |-| mtcCmdHdlr.cpp    ( 437) cmd_handler             : Info : controller-0 REBOOT Request Succeeded
2020-03-20T07:42:12.924 [2285790.01182] controller-1 mtcAgent inv mtcInvApi.cpp     ( 334) mtcInvApi_update_task   : Info : controller-0 Task: Rebooting (seq:109)
2020-03-20T07:42:12.924 [2285790.01183] controller-1 mtcAgent cmd mtcCmdHdlr.cpp    ( 459) cmd_handler             : Info : controller-0 searching for offline ; next reboot attempt in 7 seconds

BR,
-Lianhao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200320/e26661fd/attachment-0001.html>


More information about the Starlingx-discuss mailing list