March 2020 - Starlingx-discuss - lists.starlingx.io

[Starlingx-discuss] [build-report] STX_build_layer_flock_master_master - Build # 53 - Failure!
by build.starlingx＠gmail.com 21 Mar '20

21 Mar '20

Project: STX_build_layer_flock_master_master Build #: 53 Status: Failure Timestamp: 20200321T114452Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200… -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: false FORCE_BUILD: false

1 0

[Starlingx-discuss] [build-report] STX_build_pre_installer_layered - Build # 147 - Failure!
by build.starlingx＠gmail.com 21 Mar '20

21 Mar '20

Project: STX_build_pre_installer_layered Build #: 147 Status: Failure Timestamp: 20200321T121612Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200… -------------------------------------------------------------------------------- Parameters MY_WORKSPACE: /localdisk/loadbuild/jenkins/master-flock/20200321T114452Z DOCKER_BUILD_ID: jenkins-master-flock-20200321T114452Z-builder OS: centos MY_REPO: /localdisk/designer/jenkins/master-flock/cgcs-root PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200… PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/flock/20200321T114452Z/logs MASTER_JOB_NAME: STX_build_layer_flock_master_master LAYER: flock MY_REPO_ROOT: /localdisk/designer/jenkins/master-flock BUILD_ISO: true

1 0

[Starlingx-discuss] [build-report] STX_build_master_master - Build # 481 - Failure!
by build.starlingx＠gmail.com 21 Mar '20

21 Mar '20

Project: STX_build_master_master Build #: 481 Status: Failure Timestamp: 20200321T054453Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20200321T05… -------------------------------------------------------------------------------- Parameters BUILD_CONTAINERS_DEV: false BUILD_CONTAINERS_STABLE: true FORCE_BUILD: true

1 0

[Starlingx-discuss] [build-report] STX_build_post_installer - Build # 616 - Failure!
by build.starlingx＠gmail.com 21 Mar '20

21 Mar '20

Project: STX_build_post_installer Build #: 616 Status: Failure Timestamp: 20200321T095849Z Check logs at: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20200321T05… -------------------------------------------------------------------------------- Parameters MY_WORKSPACE: /localdisk/loadbuild/jenkins/master/20200321T054453Z DOCKER_BUILD_ID: jenkins-master-20200321T054453Z-builder OS: centos MY_REPO: /localdisk/designer/jenkins/master/cgcs-root PUBLISH_LOGS_URL: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20200321T05… PUBLISH_LOGS_BASE: /export/mirror/starlingx/master/centos/20200321T054453Z/logs MASTER_JOB_NAME: STX_build_master_master MY_REPO_ROOT: /localdisk/designer/jenkins/master

1 0

[Starlingx-discuss] stx docs input: 1861438, 1860995
by Ning, Antai (Andy) 20 Mar '20

20 Mar '20

Hi All, We have enhanced system's certificate configuration ability to be able to: - Install multiple CA certificates by "system certificate-install -m ssl_ca <file with multiple CA certificates>" The file may contains one or more CA certificates in X.509 PEM format. - Uninstall a particular CA certificate by its uuid by "system certificate-uninstall -m ssl_ca <UUID of the CA certificate>" The uninstall supports only CA (mode ssl_ca) certificate. The enhancement is for US1861438, US1860995. The changes are merged as reviews: https://review.opendev.org/#/c/711538<https://review.opendev.org/#/c/711538/16> https://review.opendev.org/#/c/712152<https://review.opendev.org/#/c/712152/4> https://review.opendev.org/#/c/711633<https://review.opendev.org/#/c/711633/> Please let me know if you need further information. Thanks, Andy Gerrit Code Review<https://review.opendev.org/#/c/711633/> ID: Subject: Status: Owner: Project: Branch: Updated: Size: CR: RP: V: W: 713711 [WIP] Support Redfish Auth review.opendev.org Gerrit Code Review<https://review.opendev.org/#/c/712152/4> ID: Subject: Status: Owner: Project: Branch: Updated: Size: CR: RP: V: W: 713711 [WIP] Support Redfish Auth review.opendev.org Gerrit Code Review<https://review.opendev.org/#/c/711538/16> ID: Subject: Status: Owner: Project: Branch: Updated: Size: CR: RP: V: W: 713711 [WIP] Support Redfish Auth review.opendev.org

1 1

[Starlingx-discuss] Sanity Master Test MONOLITHIC build - ISO 20200320
by Stock, Ruediger 20 Mar '20

20 Mar '20

Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2020-March-20 (http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20200320T01…) Status: GREEN =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] AIO - Duplex [BLOCKED] Setup 04 TCs [BLOCKED] Provisioning 01 TCs [BLOCKED] Sanity OpenStack 52 TCs [BLOCKED] Sanity Platform 07 TCs [BLOCKED] TOTAL: [ 64 TCs ] Standard - Local Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Standard External - Dedicated Storage (2+2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 09 TCs [PASS] TOTAL: [ 66 TCs ] =========================================== Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] AIO - Duplex [BLOCKED] Setup 04 TCs [BLOCKED] Provisioning 01 TCs [BLOCKED] Sanity OpenStack 52 TCs [BLOCKED] Sanity Platform 07 TCs [BLOCKED] TOTAL: [ 64 TCs ] Standard - Storage (2+2) Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 08 TCs [PASS] TOTAL: [ 65 TCs ] Both sanity tests on duplex hit the issue https://bugs.launchpad.net/starlingx/+bug/1856078 Regards, STX Validation Team Rüdiger Stock Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928

1 0

[Starlingx-discuss] StarlingX feedback
by Ildiko Vancsa 20 Mar '20

20 Mar '20

Hi StarlingX Community, As you may already know we have a user survey continuously open to see interest and receive feedback from people who evaluate and deploy StarlingX. I received a more detailed feedback through the survey recently that I would like to share with you below. The person who provided the feedback has good experience with the Wind River Titanium platform and that is why he decided to look into StarlingX and try it out. He marked MEC as his primary use case in the survey. With limited time and resources he had a small test environment to play with that included controller nodes on 2 VM's and 4 physical compute nodes as the following: * Controllers: * 1x HPE DL360 Gen8 (1x12core CPU, 32GB RAM) * Computes: * 2x HPE DL360 Gen9 (2x10core CPU, 128GB RAM) * 2x HPE DL380 Gen9 (2x18core CPU, 192GB RAM) Starling X versions tested: * 2.0 * 3.0 Installation experience: As it is very similar to the Titanium platform the installation of controllers and computes went pretty smooth and there weren’t too many issues. Most of the challenges came from controlling the containers in the new environment which was an expected experience. Due to the lack of documentation at that time about configuration steps the required steps were performed as experimental steps until finding the right ones. Challenges: The way how OpenStack networking was propagated over data networks turned out to be very problematic and caused hitting the wall a few times. In details you have to configure data networks over the platform management components, but then you have to set VLAN's you are going to use over data networks on the OpenStack side. This was not documented anywhere. Once data network is set up it was also a challenge to get VM’s running and connecting to networks since computes would become unstable. In summary he is not a fan of decoupling the platform parts from the OpenStack part. Issues with stability: - controllers going out of sync (maybe due to running them in VM's on the same host that was a bit limited with resources) - computes loosing Openstack helm packages after reboot - computes had random Neutron or Nova services crash when deploying VM’s Overall summary: At this point StarlingX doesn't feel stable enough for critical loads but there is a lot of potential in the platform. It is good to be able to have 2 controller nodes instead of 4 as some other OpenStack deployments demand. Also architecture choices done under the hood are quite nice and simple which he liked. Big plus is having no hassle with writing bunch of deployment templates. Platform monitoring is great and patching also looks promising but he never got the time to test it. The person who provided the feedback currently has very limited time to look into StarlingX but hoping to be able to get back to it in the future. In case you have follow up questions please provide that on this mail thread or if there’s a demand for it I will check if he would be available to join one of the community calls to follow up on the above. Thanks and Best Regards, Ildikó

2 1

Re: [Starlingx-discuss] iPXE for IPv6
by Richard, Joseph 20 Mar '20

20 Mar '20

From the tcpdump it shows that it still tries sending a boot over ipv4 after getting an ipv6 address. I don't see any indication that the controller is doing anything incorrectly, so it is probably something with the configuration on the node you are trying to bring up. What sort of config is this? Is this on hardware(which) or virtualbox(which version)? What steps are you following to setup the initial install of the node that you are trying to boot? ________________________________ From: 张鲲鹏 <zhang.kunpeng(a)99cloud.net> Sent: Monday, March 16, 2020 11:27 PM To: Richard, Joseph <Joseph.Richard(a)windriver.com> Cc: Khalil, Ghada <Ghada.Khalil(a)windriver.com>; 黄舒泉 <huang.shuquan(a)99cloud.net>; Jolliffe, Ian <Ian.Jolliffe(a)windriver.com>; Peters, Matt <Matt.Peters(a)windriver.com>; Le, Huifeng <huifeng.le(a)intel.com>; Yong Hu <yong.hu(a)intel.com> Subject: Re: [Starlingx-discuss] iPXE for IPv6 Hi Joseph, Following are the logs when boot a host from ipxe. No errors to be found. And the boot file are not sent as the tcpdump show. sysinv.log sysinv 2020-03-17 02:51:52.977 959934 INFO sysinv.cmd.dnsmasq_lease_update [-] Called 'add' for mac '00:68:27:6e:8a:42' with ip 'fd00::3f' sysinv 2020-03-17 02:51:53.217 959934 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672 sysinv 2020-03-17 02:51:53.229 95238 INFO sysinv.conductor.manager [-] receiving dhcp_lease: <sysinv.openstack.common.rpc.amqp.RpcContext object at 0x7f33cb7cf4d0> pxeboot dhcpv6 enp2s2 00:68:27:6e:8a:42 fd00::3f None sysinv 2020-03-17 02:51:53.261 95238 INFO sysinv.conductor.manager [-] Not creating ihost for mac: 00:68:27:6e:8a:42 because it already exists with uuid: 4e8ebd40-11a6-4d8e-8a8e-b2f8a703bc7e sysinv 2020-03-17 02:52:27.718 95238 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Prerequisites not met. deamon.log 2020-03-17T02:51:49.000 controller-0 collectd[81838]: info ovs interface plugin failed to initial because pid file for ovs-vswitchd doesn't exist 2020-03-17T02:51:51.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPDISCOVER(enp2s2) 52:54:00:0e:6a:02 no address available 2020-03-17T02:51:51.000 controller-0 dnsmasq-dhcp[4093358]: info RTR-SOLICIT(enp2s2) 52:54:00:0e:6a:02 2020-03-17T02:51:51.000 controller-0 dnsmasq-dhcp[4093358]: info RTR-ADVERT(enp2s2) fd00:: 2020-03-17T02:51:51.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPSOLICIT(enp2s2) 00:04:2c:7e:7b:71:b2:c5:49:df:a6:53:00:68:27:6e:8a:42 2020-03-17T02:51:51.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPADVERTISE(enp2s2) fd00::3f 00:04:2c:7e:7b:71:b2:c5:49:df:a6:53:00:68:27:6e:8a:42 2020-03-17T02:51:51.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPREQUEST(enp2s2) 00:04:2c:7e:7b:71:b2:c5:49:df:a6:53:00:68:27:6e:8a:42 2020-03-17T02:51:51.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPREPLY(enp2s2) fd00::3f 00:04:2c:7e:7b:71:b2:c5:49:df:a6:53:00:68:27:6e:8a:42 2020-03-17T02:51:52.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPDISCOVER(enp2s2) 52:54:00:0e:6a:02 no address available 2020-03-17T02:51:52.000 controller-0 dnsmasq-script[4093358]: debug sysinv 2020-03-17 02:51:52.977 959934 INFO sysinv.cmd.dnsmasq_lease_update [-] Called 'add' for mac '00:68:27:6e:8a:42' with ip 'fd00::3f'^[[00m 2020-03-17T02:51:53.000 controller-0 dnsmasq-script[4093358]: debug sysinv 2020-03-17 02:51:53.217 959934 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.204.1:5672^[[00m 2020-03-17T02:51:54.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPDISCOVER(enp2s2) 52:54:00:0e:6a:02 no address available 2020-03-17T02:51:58.000 controller-0 dnsmasq-dhcp[4093358]: info DHCPDISCOVER(enp2s2) 52:54:00:0e:6a:02 no address available tcpdump -i enp2s2 03:03:37.966202 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:38.471640 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:38.501756 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:38.977102 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:39.333447 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:0e:6a:02 (oui Unknown), length 402 03:03:39.333541 IP6 fe80::5054:ff:fe0e:6a02 > ff02::2: ICMP6, router solicitation, length 16 03:03:39.334408 IP6 fe80::5054:ff:fe5c:f3d7 > fe80::5054:ff:fe0e:6a02: ICMP6, router advertisement, length 88 03:03:39.345812 IP6 fe80::5054:ff:fe0e:6a02.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit 03:03:39.346207 IP6 fe80::5054:ff:fe5c:f3d7.dhcpv6-server > fe80::5054:ff:fe0e:6a02.dhcpv6-client: dhcp6 advertise 03:03:39.346488 IP6 fe80::5054:ff:fe0e:6a02.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 request 03:03:39.349335 IP6 fe80::5054:ff:fe5c:f3d7.dhcpv6-server > fe80::5054:ff:fe0e:6a02.dhcpv6-client: dhcp6 reply 03:03:39.482543 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:39.503651 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:39.988061 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:40.311669 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:0e:6a:02 (oui Unknown), length 402 03:03:40.493543 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:40.505647 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:40.999049 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:41.504513 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:41.507747 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:42.010033 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:42.288943 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:0e:6a:02 (oui Unknown), length 402 03:03:42.509638 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:42.515480 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:43.020834 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:43.511663 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:43.526277 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:44.031727 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:44.345728 IP6 fe80::5054:ff:fe5c:f3d7 > fe80::5054:ff:fe0e:6a02: ICMP6, neighbor solicitation, who has fe80::5054:ff:fe0e:6a02, length 32 03:03:44.345968 IP6 fe80::5054:ff:fe0e:6a02 > fe80::5054:ff:fe5c:f3d7: ICMP6, neighbor advertisement, tgt is fe80::5054:ff:fe0e:6a02, length 32 03:03:44.513797 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:44.537218 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:45.042730 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:45.515707 ARP, Request who-has controller-1 tell controller-0, length 28 03:03:45.548183 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:46.053802 IP controller-0.51259 > 239.1.1.2.mzap: UDP, length 75 03:03:46.243493 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:0e:6a:02 (oui Unknown), length 402 03:03:46.517640 ARP, Request who-has controller-1 tell controller-0, length 28 在 2020年3月13日，06:10，Richard, Joseph <Joseph.Richard(a)windriver.com<mailto:Joseph.Richard@windriver.com>> 写道： <PastedGraphic-3.png><PastedGraphic-2.png><PastedGraphic-1.jpeg>_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss(a)lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

1 0

[Starlingx-discuss] [Python3] Analysis of nfv using python3
by An, Ran1 20 Mar '20

20 Mar '20

Hi all: I’m working on task “upgrade project nfv to support python3 runtime env”. By analyzing and testing, there will be 32 new python3 rpms, which may impact current system. · 16 new rpms from official centos7 repo; · 12 new stx-rpms rebuilt with srpms from official centos7 repo (turn on python3 supported switch in spec); · 4 stx-rpms built from tar.gz files. Details including pkg dependency trees and current status could be found in [1] With patches [2], we could get an iso with nfv services (vim,vim-api, vim-webserver) running in python3. However, there is an run-time issues. Model “fm_api” in python3 is required to support vim service reporting alarms/event_logs to fault manager (service fm-mgr). It brings following issues: 1. to build python3 "fm-api" rpm, pkg “python3-rpm-generators” is required. But it is conflict with pkg “rpm-build” built in the compile layer, because both of them have file (/usr/lib/rpm/pythondistdeps.py). 2. as other flock services, like sysinv, are still running in python2, so it require fm-api to both run in python2 env and python3 env. It introduces a new topic: ensure it is safe to run python2-fm-api and python3-fm-api at the same time. The scope of this task seems expand. Is there any suggestions? [1] https://bugs.launchpad.net/starlingx/+bug/1808073/+attachment/5335996/+file… [2] patches: https://review.opendev.org/709958; https://review.opendev.org/709960; https://review.opendev.org/709983/1 ; https://review.opendev.org/709980 Thanks Ran

1 1

[Starlingx-discuss] STX3 heartbeat lost causing controller and worker nodes rebooting
by Lu, Lianhao 20 Mar '20

20 Mar '20

Hi STX gurus, Currently I met an issue of standby controller and worker nodes keep rebooting due to the lost of heartbeat in STX3, hope someone could give me some help. Thanks very much! My system contains the following configuration: Controllers: Two Inspur nf8260m5 servers each with 2 Intel X550T NICs, 1 X550 NIC for OAM network and the other for mgmt network. Worker node: one x86_64 server with 2 Intel X722 NICs, 1 X722 NIC for mgmt. network and the other for data. The 2 OAM network NICs are all connected to a switch, and the 3 mgmt NICs are connected to another switch. I follow the STX installation guide of the deployment option Standard with Controller Storage, but now I'm facing an issue of the controller-0(stand-by controller) and worker-0(the only work node) keeps rebooting. Though I can successfully ping to controller-0 and worker-0 from my active controller-1, and vice versa. I still see the following log in /var/log/mtcAgent.log. ------ log excerpt from mtcAgent.log ----- 2020-03-20T07:42:02.837 [2285790.01035] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor set (Mgmnt) 2020-03-20T07:42:02.837 [2285790.01036] controller-1 mtcAgent --- mtcNodeMnfa.cpp ( 131) mnfa_add_host : Info : controller-0 MNFA new candidate (Mgmnt) 1 enabled hosts (threshold:2) (1:Mgmnt:10) (0:Clstr:0) 2020-03-20T07:42:03.047 [2285790.01037] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade set (Mgmnt) 2020-03-20T07:42:03.047 [2285790.01038] controller-1 mtcAgent hbs nodeClass.cpp (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss *** 2020-03-20T07:42:03.047 [2285790.01039] controller-1 mtcAgent inv mtcInvApi.cpp (1119) mtcInvApi_update_state : Info : controller-0 degraded (seq:97) 2020-03-20T07:42:03.468 [2285790.01040] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat loss (Mgmnt) 2020-03-20T07:42:03.468 [2285790.01041] controller-1 mtcAgent hbs nodeClass.cpp (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss *** 2020-03-20T07:42:03.468 [2285790.01042] controller-1 mtcAgent hbs nodeClass.cpp (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure 2020-03-20T07:42:03.468 [2285790.01043] controller-1 mtcAgent inv mtcInvApi.cpp (1119) mtcInvApi_update_state : Info : controller-0 failed (seq:98) 2020-03-20T07:42:03.468 [2285790.01044] controller-1 mtcAgent hbs nodeClass.cpp (4965) manage_heartbeat_failure: Warn : controller-0 starting graceful recovery 2020-03-20T07:42:03.468 [2285790.01045] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (1643) recovery_handler : Info : controller-0 Graceful Recovery (uptime was 473) 2020-03-20T07:42:03.468 [2285790.01046] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-0 hbsAgent 2020-03-20T07:42:03.468 [2285790.01047] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-1 hbsAgent 2020-03-20T07:42:03.468 [2285790.01048] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1684) recovery_handler : Warn : controller-0 Graceful Recovery (1 of 3) 2020-03-20T07:42:03.468 [2285790.01049] controller-1 mtcAgent inv mtcInvApi.cpp ( 334) mtcInvApi_update_task : Info : controller-0 Task: Graceful Recovery (seq:99) 2020-03-20T07:42:03.468 [2285790.01050] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:03.468 [2285790.01051] controller-1 mtcAgent hbs nodeClass.cpp (4833) hbs_minor_clear : Warn : controller-0 MNFA host tally (Mgmnt:0) 2020-03-20T07:42:03.468 [2285790.01052] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:05.473 [2285790.01053] controller-1 mtcAgent |-| nodeClass.cpp (2542) start_offline_handler : Info : controller-0 starting offline handler (unlocked-enabled-failed) (stage:0) 2020-03-20T07:42:05.473 [2285790.01054] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1729) recovery_handler : Info : controller-0 requesting mtcAlive with 5 sec timeout 2020-03-20T07:42:05.473 [2285790.01055] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (3242) offline_handler : Info : controller-0 starting 100 msec offline audit (enabled-failed) 2020-03-20T07:42:05.473 [2285790.01056] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1747) recovery_handler : Info : controller-0 got requested mtcAlive 2020-03-20T07:42:05.473 [2285790.01057] controller-1 mtcAgent |-| nodeClass.cpp (2555) stop_offline_handler : Info : controller-0 stopping offline handler (unlocked-enabled-failed) (stage:3) 2020-03-20T07:42:05.473 [2285790.01058] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1814) recovery_handler : Warn : controller-0 Connectivity Recovered ; host did not reset 2020-03-20T07:42:05.473 [2285790.01059] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1816) recovery_handler : Warn : controller-0 ... continuing with graceful recovery 2020-03-20T07:42:05.473 [2285790.01060] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1817) recovery_handler : Warn : controller-0 ... with no affect to host services 2020-03-20T07:42:05.473 [2285790.01061] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 start host service sent to controller-0 hbsAgent 2020-03-20T07:42:05.473 [2285790.01062] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 start host service sent to controller-1 hbsAgent 2020-03-20T07:42:05.473 [2285790.01063] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (2489) recovery_handler : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event) 2020-03-20T07:42:05.474 [2285790.01064] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:05.474 [2285790.01065] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:05.985 [2285790.01066] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor set (Mgmnt) 2020-03-20T07:42:05.985 [2285790.01067] controller-1 mtcAgent --- mtcNodeMnfa.cpp ( 131) mnfa_add_host : Info : controller-0 MNFA new candidate (Mgmnt) 1 enabled hosts (threshold:2) (1:Mgmnt:11) (0:Clstr:0) 2020-03-20T07:42:06.196 [2285790.01068] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade set (Mgmnt) 2020-03-20T07:42:06.196 [2285790.01069] controller-1 mtcAgent hbs nodeClass.cpp (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss *** 2020-03-20T07:42:06.616 [2285790.01070] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat loss (Mgmnt) 2020-03-20T07:42:06.616 [2285790.01071] controller-1 mtcAgent hbs nodeClass.cpp (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss *** 2020-03-20T07:42:06.616 [2285790.01072] controller-1 mtcAgent hbs nodeClass.cpp (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure 2020-03-20T07:42:06.616 [2285790.01073] controller-1 mtcAgent hbs nodeClass.cpp (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery 2020-03-20T07:42:06.616 [2285790.01074] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (1643) recovery_handler : Info : controller-0 Graceful Recovery (uptime was 478) 2020-03-20T07:42:06.616 [2285790.01075] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-0 hbsAgent 2020-03-20T07:42:06.616 [2285790.01076] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-1 hbsAgent 2020-03-20T07:42:06.616 [2285790.01077] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1684) recovery_handler : Warn : controller-0 Graceful Recovery (2 of 3) 2020-03-20T07:42:06.616 [2285790.01078] controller-1 mtcAgent inv mtcInvApi.cpp ( 334) mtcInvApi_update_task : Info : controller-0 Task: Graceful Recovery Retry (seq:100) 2020-03-20T07:42:06.616 [2285790.01079] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:06.616 [2285790.01080] controller-1 mtcAgent hbs nodeClass.cpp (4833) hbs_minor_clear : Warn : controller-0 MNFA host tally (Mgmnt:0) 2020-03-20T07:42:06.616 [2285790.01081] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:08.569 [2285790.01082] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (1421) enable_handler : Info : worker-0 heartbeating 2020-03-20T07:42:08.574 [2285790.01083] controller-1 mtcAgent inv mtcInvApi.cpp ( 437) mtcInvApi_force_task : Info : worker-0 task clear (seq:96) (was:Enabling) 2020-03-20T07:42:08.574 fmAPI.cpp(490): Enqueue raise alarm request: UUID (b24f0d21-6c0d-4862-99de-6e8ad1f04b99) alarm id (200.022) instant id (host=worker-0.state=enabled) 2020-03-20T07:42:08.574 [2285790.01084] controller-1 mtcAgent hbs nodeClass.cpp (6188) allStateChange : Info : worker-0 unlocked-enabled-degraded (seq:97) 2020-03-20T07:42:08.605 fmAlarmUtils.cpp(624): Sending FM raise alarm request: alarm_id (200.022), entity_id (host=worker-0.state=enabled) 2020-03-20T07:42:08.620 [2285790.01085] controller-1 mtcAgent |-| nodeClass.cpp (2542) start_offline_handler : Info : controller-0 starting offline handler (unlocked-enabled-failed) (stage:0) 2020-03-20T07:42:08.620 [2285790.01086] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1729) recovery_handler : Info : controller-0 requesting mtcAlive with 5 sec timeout 2020-03-20T07:42:08.620 [2285790.01087] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (3242) offline_handler : Info : controller-0 starting 100 msec offline audit (enabled-failed) 2020-03-20T07:42:08.620 [2285790.01088] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1747) recovery_handler : Info : controller-0 got requested mtcAlive 2020-03-20T07:42:08.620 [2285790.01089] controller-1 mtcAgent |-| nodeClass.cpp (2555) stop_offline_handler : Info : controller-0 stopping offline handler (unlocked-enabled-failed) (stage:3) 2020-03-20T07:42:08.620 [2285790.01090] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1814) recovery_handler : Warn : controller-0 Connectivity Recovered ; host did not reset 2020-03-20T07:42:08.620 [2285790.01091] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1816) recovery_handler : Warn : controller-0 ... continuing with graceful recovery 2020-03-20T07:42:08.620 [2285790.01092] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1817) recovery_handler : Warn : controller-0 ... with no affect to host services 2020-03-20T07:42:08.621 [2285790.01093] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 start host service sent to controller-0 hbsAgent 2020-03-20T07:42:08.621 [2285790.01094] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 start host service sent to controller-1 hbsAgent 2020-03-20T07:42:08.621 [2285790.01095] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (2489) recovery_handler : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event) 2020-03-20T07:42:08.621 [2285790.01096] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:09.134 [2285790.01104] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor set (Mgmnt) 2020-03-20T07:42:09.134 [2285790.01105] controller-1 mtcAgent --- mtcNodeMnfa.cpp ( 131) mnfa_add_host : Info : controller-0 MNFA new candidate (Mgmnt) 2 enabled hosts (threshold:2) (1:Mgmnt:12) (0:Clstr:0) 2020-03-20T07:42:09.344 [2285790.01106] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade set (Mgmnt) 2020-03-20T07:42:09.344 [2285790.01107] controller-1 mtcAgent hbs nodeClass.cpp (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss *** 2020-03-20T07:42:09.764 [2285790.01108] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat loss (Mgmnt) 2020-03-20T07:42:09.764 [2285790.01109] controller-1 mtcAgent hbs nodeClass.cpp (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss *** 2020-03-20T07:42:09.764 [2285790.01110] controller-1 mtcAgent hbs nodeClass.cpp (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure 2020-03-20T07:42:09.764 [2285790.01111] controller-1 mtcAgent hbs nodeClass.cpp (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery 2020-03-20T07:42:09.765 [2285790.01112] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (1643) recovery_handler : Info : controller-0 Graceful Recovery (uptime was 481) 2020-03-20T07:42:09.765 [2285790.01113] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-0 hbsAgent 2020-03-20T07:42:09.765 [2285790.01114] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-1 hbsAgent 2020-03-20T07:42:09.765 [2285790.01115] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1684) recovery_handler : Warn : controller-0 Graceful Recovery (3 of 3) 2020-03-20T07:42:09.765 [2285790.01116] controller-1 mtcAgent inv mtcInvApi.cpp ( 334) mtcInvApi_update_task : Info : controller-0 Task: Graceful Recovery Retry (seq:101) 2020-03-20T07:42:09.765 [2285790.01117] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:09.765 [2285790.01118] controller-1 mtcAgent hbs nodeClass.cpp (4833) hbs_minor_clear : Warn : controller-0 MNFA host tally (Mgmnt:0) 2020-03-20T07:42:09.765 [2285790.01119] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:10.468 [2285790.01120] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (3473) online_handler : Info : controller-0 mtcAlive ; going 'online' 2020-03-20T07:42:10.468 [2285790.01121] controller-1 mtcAgent inv mtcInvApi.cpp (1119) mtcInvApi_update_state : Info : controller-0 online (seq:102) 2020-03-20T07:42:11.770 [2285790.01122] controller-1 mtcAgent |-| nodeClass.cpp (2542) start_offline_handler : Info : controller-0 starting offline handler (unlocked-enabled-online) (stage:0) 2020-03-20T07:42:11.770 [2285790.01123] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1729) recovery_handler : Info : controller-0 requesting mtcAlive with 5 sec timeout 2020-03-20T07:42:11.770 [2285790.01124] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (3242) offline_handler : Info : controller-0 starting 100 msec offline audit (enabled-online) 2020-03-20T07:42:11.770 [2285790.01125] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1747) recovery_handler : Info : controller-0 got requested mtcAlive 2020-03-20T07:42:11.770 [2285790.01126] controller-1 mtcAgent |-| nodeClass.cpp (2555) stop_offline_handler : Info : controller-0 stopping offline handler (unlocked-enabled-online) (stage:3) 2020-03-20T07:42:11.770 [2285790.01127] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1814) recovery_handler : Warn : controller-0 Connectivity Recovered ; host did not reset 2020-03-20T07:42:11.770 [2285790.01128] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1816) recovery_handler : Warn : controller-0 ... continuing with graceful recovery 2020-03-20T07:42:11.770 [2285790.01129] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1817) recovery_handler : Warn : controller-0 ... with no affect to host services 2020-03-20T07:42:11.770 [2285790.01130] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 start host service sent to controller-0 hbsAgent 2020-03-20T07:42:11.770 [2285790.01131] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 start host service sent to controller-1 hbsAgent 2020-03-20T07:42:11.770 [2285790.01132] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (2489) recovery_handler : Info : controller-0 Starting 11 sec Heartbeat Soak (with ready event) 2020-03-20T07:42:11.770 [2285790.01133] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:11.770 [2285790.01134] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:12.282 [2285790.01135] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor set (Mgmnt) 2020-03-20T07:42:12.282 [2285790.01136] controller-1 mtcAgent --- mtcNodeMnfa.cpp ( 131) mnfa_add_host : Info : controller-0 MNFA new candidate (Mgmnt) 2 enabled hosts (threshold:2) (1:Mgmnt:13) (0:Clstr:0) 2020-03-20T07:42:12.493 [2285790.01137] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade set (Mgmnt) 2020-03-20T07:42:12.493 [2285790.01138] controller-1 mtcAgent hbs nodeClass.cpp (5060) manage_heartbeat_degrade: Warn : controller-0 Mgmnt *** Heartbeat Miss *** 2020-03-20T07:42:12.913 [2285790.01139] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat loss (Mgmnt) 2020-03-20T07:42:12.913 [2285790.01140] controller-1 mtcAgent hbs nodeClass.cpp (4941) manage_heartbeat_failure:Error : controller-0 Mgmnt *** Heartbeat Loss *** 2020-03-20T07:42:12.913 [2285790.01141] controller-1 mtcAgent hbs nodeClass.cpp (4952) manage_heartbeat_failure:Error : controller-0 Mgmnt network heartbeat failure 2020-03-20T07:42:12.913 [2285790.01142] controller-1 mtcAgent inv mtcInvApi.cpp (1119) mtcInvApi_update_state : Info : controller-0 failed (seq:103) 2020-03-20T07:42:12.914 [2285790.01143] controller-1 mtcAgent hbs nodeClass.cpp (4961) manage_heartbeat_failure: Warn : controller-0 restarting graceful recovery 2020-03-20T07:42:12.914 [2285790.01144] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp (1643) recovery_handler : Info : controller-0 Graceful Recovery (uptime was 484) 2020-03-20T07:42:12.914 [2285790.01145] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-0 hbsAgent 2020-03-20T07:42:12.914 [2285790.01146] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-1 hbsAgent 2020-03-20T07:42:12.914 [2285790.01147] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (1671) recovery_handler :Error : controller-0 Graceful Recovery Failed (retries=4) 2020-03-20T07:42:12.914 [2285790.01148] controller-1 mtcAgent |-| nodeClass.cpp (7590) force_full_enable : Info : controller-0 Forcing Full Enable Sequence 2020-03-20T07:42:12.914 [2285790.01149] controller-1 mtcAgent hbs nodeClass.cpp (1681) alarm_enabled_failure :Error : controller-0 critical enable failure 2020-03-20T07:42:12.914 [2285790.01150] controller-1 mtcAgent alm mtcAlarm.cpp ( 417) mtcAlarm_critical :Error : controller-0 setting critical 'In-Service' failure alarm (200.004 ) 2020-03-20T07:42:12.914 fmAPI.cpp(490): Enqueue raise alarm request: UUID (4f1c52ab-098d-4b87-aafc-acfdd2c7ac7a) alarm id (200.004) instant id (host=controller-0) 2020-03-20T07:42:12.914 fmAPI.cpp(490): Enqueue raise alarm request: UUID (d6a99af3-2e3b-4bf4-b0db-07f966fafea2) alarm id (200.022) instant id (host=controller-0.state=disabled) 2020-03-20T07:42:12.914 [2285790.01151] controller-1 mtcAgent hbs nodeClass.cpp (6188) allStateChange : Info : controller-0 unlocked-disabled-failed (seq:104) 2020-03-20T07:42:12.914 [2285790.01152] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:12.914 [2285790.01153] controller-1 mtcAgent hbs nodeClass.cpp (4833) hbs_minor_clear : Warn : controller-0 MNFA host tally (Mgmnt:0) 2020-03-20T07:42:12.914 [2285790.01154] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp ( 573) enable_handler :Error : controller-0 Main Enable FSM (from failed) 2020-03-20T07:42:12.914 [2285790.01155] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-0 hbsAgent 2020-03-20T07:42:12.914 [2285790.01156] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-1 hbsAgent 2020-03-20T07:42:12.914 [2285790.01157] controller-1 mtcAgent vim mtcVimApi.cpp ( 251) mtcVimApi_state_change :Error : controller-0 {"state-change": {"administrative":"unlocked","operational":"disabled","availability":"failed","subfunction_oper":"disabled","subfunction_avail":"not-installed"},"hostname":"controller-0","uuid":"28412a2d-fb95-472d-9797-c2ef4b18752f","subfunctions":"controller","personality":"controller"} 2020-03-20T07:42:12.914 [2285790.01158] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:12.914 [2285790.01159] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:12.914 [2285790.01160] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp ( 769) enable_handler : Info : controller-0 Main Enable FSM (from start) 2020-03-20T07:42:12.914 [2285790.01161] controller-1 mtcAgent --- mtcWorkQueue.cpp ( 750) workQueue_purge : Warn : controller-0 purging 4 items from workQueue 2020-03-20T07:42:12.914 [2285790.01162] controller-1 mtcAgent --- mtcWorkQueue.cpp ( 764) workQueue_purge : Warn : controller-0 mtcInvApi_update_state seq:103 ... did not complete (Wait) 2020-03-20T07:42:12.914 [2285790.01163] controller-1 mtcAgent --- mtcWorkQueue.cpp ( 758) workQueue_purge : Warn : controller-0 mtcInvApi_force_states seq:104 ... was not executed 2020-03-20T07:42:12.914 [2285790.01164] controller-1 mtcAgent --- mtcWorkQueue.cpp ( 758) workQueue_purge : Warn : controller-0 mtcInvApi_force_states seq:105 ... was not executed 2020-03-20T07:42:12.914 [2285790.01165] controller-1 mtcAgent --- mtcWorkQueue.cpp ( 758) workQueue_purge : Warn : controller-0 mtcVimApi_state_change seq:106 ... was not executed 2020-03-20T07:42:12.914 [2285790.01166] controller-1 mtcAgent mgr mtcSmgrApi.cpp ( 209) mtcSmgrApi_request : Info : controller-0 sending 'unlocked-disabled' request to HA Service Manager 2020-03-20T07:42:12.919 [2285790.01167] controller-1 mtcAgent mgr mtcSmgrApi.cpp ( 300) mtcSmgrApi_request : Info : controller-0 is 'disabled' to Service Management 2020-03-20T07:42:12.919 [2285790.01168] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:12.919 [2285790.01169] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-0 hbsAgent 2020-03-20T07:42:12.919 [2285790.01170] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 878) send_hbs_command : Info : controller-0 stop host service sent to controller-1 hbsAgent 2020-03-20T07:42:12.919 [2285790.01171] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat minor clear (Mgmnt) 2020-03-20T07:42:12.919 [2285790.01172] controller-1 mtcAgent |-| mtcNodeHdlrs.cpp ( 963) enable_handler : Info : controller-0 reboot 2020-03-20T07:42:12.919 [2285790.01173] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp ( 91) calc_reset_prog_timeout : Info : controller-0 Reboot/Reset progression has 40 sec 'wait for offline' timeout 2020-03-20T07:42:12.919 [2285790.01174] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp ( 95) calc_reset_prog_timeout : Info : controller-0 ... sources - mgmnt:Yes clstr:No bmc:No 2020-03-20T07:42:12.919 [2285790.01175] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 971) service_events : Info : controller-0 heartbeat degrade clear (Mgmnt) 2020-03-20T07:42:12.924 [2285790.01176] controller-1 mtcAgent inv mtcInvApi.cpp ( 334) mtcInvApi_update_task : Info : controller-0 Task: Reboot Request (seq:108) 2020-03-20T07:42:12.924 [2285790.01177] controller-1 mtcAgent |-| nodeClass.cpp (2542) start_offline_handler : Info : controller-0 starting offline handler (unlocked-disabled-failed) (stage:0) 2020-03-20T07:42:12.924 [2285790.01178] controller-1 mtcAgent hdl mtcNodeHdlrs.cpp (3242) offline_handler : Info : controller-0 starting 100 msec offline audit (disabled-failed) 2020-03-20T07:42:12.924 [2285790.01179] controller-1 mtcAgent msg mtcCtrlMsg.cpp ( 629) send_mtc_cmd : Info : controller-0 sending 'reboot' request (Mgmnt network) 2020-03-20T07:42:12.924 [2285790.01180] controller-1 mtcAgent cmd mtcCmdHdlr.cpp ( 403) cmd_handler : Info : controller-0 waiting for REBOOT ACK 2020-03-20T07:42:12.924 [2285790.01181] controller-1 mtcAgent |-| mtcCmdHdlr.cpp ( 437) cmd_handler : Info : controller-0 REBOOT Request Succeeded 2020-03-20T07:42:12.924 [2285790.01182] controller-1 mtcAgent inv mtcInvApi.cpp ( 334) mtcInvApi_update_task : Info : controller-0 Task: Rebooting (seq:109) 2020-03-20T07:42:12.924 [2285790.01183] controller-1 mtcAgent cmd mtcCmdHdlr.cpp ( 459) cmd_handler : Info : controller-0 searching for offline ; next reboot attempt in 7 seconds BR, -Lianhao

1 0