[Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'"
---------- Forwarded message --------- 보낸사람: 오재욱 <kyle.oh95@gmail.com> Date: 2019년 3월 26일 (화) 오전 11:58 Subject: Re: [Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" To: Hu, Yong <yong.hu@intel.com> Hi Yong, Umm I think the controller0 was completely installed, but the controller1 was not. When I unlocked the controller1, fm alarm shows the error. So it does mean the controller1 was not completely installed. If so, do you think there is a reason for the installation not being completed elsewhere? Thank you for help, BR, Jaewook. 2019년 3월 26일 (화) 오전 11:53, Hu, Yong <yong.hu@intel.com>님이 작성:
Have you installed 2 controllers already?
If by far you only setup 1 controller, this alarm seems reasonable because it expects to communicate with another controller through mgt network.
Regards,
Yong
*From: *오재욱 <kyle.oh95@gmail.com> *Date: *Tuesday, 26 March 2019 at 10:45 AM *To: *starlingx-discuss <starlingx-discuss@lists.starlingx.io> *Subject: *[Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'"
Hello StarlingX Teams,
I finally deployed duplex starlingx model on two physical servers.
However, it was not truly deployed. The "all-in-one controller-1" has deploy failed,
The fault management alarm shows,
200.005
Failure: <hostname> is experiencing a persistent critical 'Management Network' communication failure."
And I think above fault is the main reason of deploy failed
I followed BIOS Setup for all nodes,
and used Feb 22 iso images.
I have no idea what can I do :(
Any help would be very appreciated.
Best Regards,
Jaewook.
================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea
-- ================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea
Hi Jaewook: Could you run ‘system host-list’ or ‘system host-show controller-1’ to see the status of controller-1? Is it available degrade or others ? Thanks. BR Austin Sun. From: 오재욱 [mailto:kyle.oh95@gmail.com] Sent: Tuesday, March 26, 2019 10:59 AM To: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" ---------- Forwarded message --------- 보낸사람: 오재욱 <kyle.oh95@gmail.com<mailto:kyle.oh95@gmail.com>> Date: 2019년 3월 26일 (화) 오전 11:58 Subject: Re: [Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" To: Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> Hi Yong, Umm I think the controller0 was completely installed, but the controller1 was not. When I unlocked the controller1, fm alarm shows the error. So it does mean the controller1 was not completely installed. If so, do you think there is a reason for the installation not being completed elsewhere? Thank you for help, BR, Jaewook. 2019년 3월 26일 (화) 오전 11:53, Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>님이 작성: Have you installed 2 controllers already? If by far you only setup 1 controller, this alarm seems reasonable because it expects to communicate with another controller through mgt network. Regards, Yong From: 오재욱 <kyle.oh95@gmail.com<mailto:kyle.oh95@gmail.com>> Date: Tuesday, 26 March 2019 at 10:45 AM To: starlingx-discuss <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" Hello StarlingX Teams, I finally deployed duplex starlingx model on two physical servers. However, it was not truly deployed. The "all-in-one controller-1" has deploy failed, The fault management alarm shows, 200.005 Failure: <hostname> is experiencing a persistent critical 'Management Network' communication failure." And I think above fault is the main reason of deploy failed I followed BIOS Setup for all nodes, and used Feb 22 iso images. I have no idea what can I do :( Any help would be very appreciated. Best Regards, Jaewook. ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea -- ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea
Jaewook, I just now looked through your puppet log, and there was no error message at all. Though the alarm was about “management network” communication failure, the essence of problem might be just about controller-1 was not under a right state. Anyway, we guess maybe something wrong with ovs on your controller-1 and it led controller-1 unlock failure. You can have a check with these stuffs: 1. Check if any errors are in “/var/log/openvswitch/ovs-vswitchd.log” in controller-1 2. Run “sudo ovs-vsctl show” on controller-1 Regards, Yong From: "Sun, Austin" <austin.sun@intel.com> Date: Tuesday, 26 March 2019 at 11:31 AM To: 오재욱 <kyle.oh95@gmail.com>, starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" Hi Jaewook: Could you run ‘system host-list’ or ‘system host-show controller-1’ to see the status of controller-1? Is it available degrade or others ? Thanks. BR Austin Sun. From: 오재욱 [mailto:kyle.oh95@gmail.com] Sent: Tuesday, March 26, 2019 10:59 AM To: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" ---------- Forwarded message --------- 보낸사람: 오재욱 <kyle.oh95@gmail.com<mailto:kyle.oh95@gmail.com>> Date: 2019년 3월 26일 (화) 오전 11:58 Subject: Re: [Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" To: Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> Hi Yong, Umm I think the controller0 was completely installed, but the controller1 was not. When I unlocked the controller1, fm alarm shows the error. So it does mean the controller1 was not completely installed. If so, do you think there is a reason for the installation not being completed elsewhere? Thank you for help, BR, Jaewook. 2019년 3월 26일 (화) 오전 11:53, Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>님이 작성: Have you installed 2 controllers already? If by far you only setup 1 controller, this alarm seems reasonable because it expects to communicate with another controller through mgt network. Regards, Yong From: 오재욱 <kyle.oh95@gmail.com<mailto:kyle.oh95@gmail.com>> Date: Tuesday, 26 March 2019 at 10:45 AM To: starlingx-discuss <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" Hello StarlingX Teams, I finally deployed duplex starlingx model on two physical servers. However, it was not truly deployed. The "all-in-one controller-1" has deploy failed, The fault management alarm shows, 200.005 Failure: <hostname> is experiencing a persistent critical 'Management Network' communication failure." And I think above fault is the main reason of deploy failed I followed BIOS Setup for all nodes, and used Feb 22 iso images. I have no idea what can I do :( Any help would be very appreciated. Best Regards, Jaewook. ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea -- ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea
Thanks Yong, 1. checked my ovs-vswitchd.log, and I think br-int has some problem. I found something weird in the log, "2019-03-25T06:11:01.884Z|00192|rconn|WARN| br-int<->tcp:127.0.0.1:6633: connection failed (Connection refused)" Is this the main reason of the problem? 2. And below is my ovs-vsctl show result: controller-1:~$ sudo ovs-vsctl show 10fb65fb-9bac-4ef2-9221-8c27f53afcd3 Manager "ptcp:6639:127.0.0.1" is_connected: true Bridge "br-phy0" Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "br-phy0" Interface "br-phy0" type: internal Port "phy-br-phy0" Interface "phy-br-phy0" type: patch options: {peer="int-br-phy0"} Port "lldp6b8235ce-e2" Interface "lldp6b8235ce-e2" type: internal Port "eth0" Interface "eth0" type: dpdk options: {dpdk-devargs="0000:81:00.0", n_rxq="2"} Bridge br-int Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port br-int Interface br-int type: internal Port "int-br-phy0" Interface "int-br-phy0" type: patch options: {peer="phy-br-phy0"} ovs_version: "2.9.0" BR, Jaewook. 2019년 3월 26일 (화) 오후 12:48, Hu, Yong <yong.hu@intel.com>님이 작성:
Jaewook,
I just now looked through your puppet log, and there was no error message at all.
Though the alarm was about “management network” communication failure, the essence of problem might be just about controller-1 was not under a right state.
Anyway, we guess maybe something wrong with ovs on your controller-1 and it led controller-1 unlock failure.
You can have a check with these stuffs:
1. Check if any errors are in “/var/log/openvswitch/ovs-vswitchd.log” in controller-1 2. Run “sudo ovs-vsctl show” on controller-1
Regards,
Yong
*From: *"Sun, Austin" <austin.sun@intel.com> *Date: *Tuesday, 26 March 2019 at 11:31 AM *To: *오재욱 <kyle.oh95@gmail.com>, starlingx-discuss < starlingx-discuss@lists.starlingx.io> *Subject: *Re: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'"
Hi Jaewook:
Could you run ‘system host-list’ or ‘system host-show controller-1’ to see the status of controller-1? Is it available degrade or others ?
Thanks.
BR Austin Sun.
That alarm is indication that Maintenance is failing Heartbeat over the Management network. Please check your network and maybe specifically its support for multicast messaging. If you lock controller-1 can you ping it from the active controller. If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files You can always run ‘collect’ or ‘collect all’ on the active controller to get a summary of the system state and use those logs for off system debug of your issue. Eric MacDonald. From: Hu, Yong [mailto:yong.hu@intel.com] Sent: Monday, March 25, 2019 11:48 PM To: Sun, Austin; 오재욱; starlingx-discuss Subject: Re: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" Jaewook, I just now looked through your puppet log, and there was no error message at all. Though the alarm was about “management network” communication failure, the essence of problem might be just about controller-1 was not under a right state. Anyway, we guess maybe something wrong with ovs on your controller-1 and it led controller-1 unlock failure. You can have a check with these stuffs: 1. Check if any errors are in “/var/log/openvswitch/ovs-vswitchd.log” in controller-1 2. Run “sudo ovs-vsctl show” on controller-1 Regards, Yong From: "Sun, Austin" <austin.sun@intel.com> Date: Tuesday, 26 March 2019 at 11:31 AM To: 오재욱 <kyle.oh95@gmail.com>, starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" Hi Jaewook: Could you run ‘system host-list’ or ‘system host-show controller-1’ to see the status of controller-1? Is it available degrade or others ? Thanks. BR Austin Sun. From: 오재욱 [mailto:kyle.oh95@gmail.com] Sent: Tuesday, March 26, 2019 10:59 AM To: starlingx-discuss <starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" ---------- Forwarded message --------- 보낸사람: 오재욱 <kyle.oh95@gmail.com<mailto:kyle.oh95@gmail.com>> Date: 2019년 3월 26일 (화) 오전 11:58 Subject: Re: [Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" To: Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>> Hi Yong, Umm I think the controller0 was completely installed, but the controller1 was not. When I unlocked the controller1, fm alarm shows the error. So it does mean the controller1 was not completely installed. If so, do you think there is a reason for the installation not being completed elsewhere? Thank you for help, BR, Jaewook. 2019년 3월 26일 (화) 오전 11:53, Hu, Yong <yong.hu@intel.com<mailto:yong.hu@intel.com>>님이 작성: Have you installed 2 controllers already? If by far you only setup 1 controller, this alarm seems reasonable because it expects to communicate with another controller through mgt network. Regards, Yong From: 오재욱 <kyle.oh95@gmail.com<mailto:kyle.oh95@gmail.com>> Date: Tuesday, 26 March 2019 at 10:45 AM To: starlingx-discuss <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" Hello StarlingX Teams, I finally deployed duplex starlingx model on two physical servers. However, it was not truly deployed. The "all-in-one controller-1" has deploy failed, The fault management alarm shows, 200.005 Failure: <hostname> is experiencing a persistent critical 'Management Network' communication failure." And I think above fault is the main reason of deploy failed I followed BIOS Setup for all nodes, and used Feb 22 iso images. I have no idea what can I do :( Any help would be very appreciated. Best Regards, Jaewook. ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea -- ================================================ Jaewook Oh (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea
Hello Eric, thanks for advice. I commented my answer to your advices, 0. Please check your network and maybe specifically its support for multicast messaging. -> Could you tell me how can I check that my network supports multicast messaging? Do I need function like 'omping'? 1. If you lock controller-1 can you ping it from the active controller. -> Yes, I can ping to controller-1 from the active one, with controller-1's management ip (192.168.204.4), and controller-1's oam ip (10.10.10.4). 2. If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files -> I attached those files :) 3. You can always run ‘collect’ or ‘collect all’ on the active controller to get a summary of the system state and use those logs for off system debug of your issue. -> Thanks, I totally didn't know about those commands! 2019년 3월 26일 (화) 오후 10:45, MacDonald, Eric <Eric.MacDonald@windriver.com>님이 작성:
That alarm is indication that Maintenance is failing Heartbeat over the Management network.
Please check your network and maybe specifically its support for multicast messaging.
If you lock controller-1 can you ping it from the active controller.
If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files
You can always run ‘collect’ or ‘collect all’ on the active controller to get a summary of the system state and use those logs for off system debug of your issue.
Eric MacDonald.
There appears to be something wrong with your multi-cast setup as the hbsAgent is not able to setup its pulse messaging socket for the management interface. hbsAgent.log – This log seems to have rotated due to successive socket resetup retries. 2019-03-27T10:33:55.984 [4194.3824110] controller-0 hbsAgent --- msgClass.cpp (1033) msgClassTx : Info : Creating controller-0 socket on port 2106 with address: 239.1.1.2 2019-03-27T10:33:55.984 [4194.3824111] controller-0 hbsAgent --- msgClass.cpp (1098) initSocket :Error : Failed to bind socket to address (22:Invalid argument) 2019-03-27T10:33:55.984 [4194.3824112] controller-0 hbsAgent hbs hbsAgent.cpp ( 575) _setup_pulse_messaging :Error : Cannot open multicast transmit socket - rc:34 (22:Invalid argument) mtcAgent.log shows that your management interface is bouncing which is causing command failures. 2019-03-22T16:57:19.449 [94960.00107] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down 2019-03-22T16:57:19.449 [94960.00108] controller-0 mtcAgent hbs nodeClass.cpp (4301) service_netlink_events : Warn : eno2 is down (oper:down) 2019-03-22T16:57:19.881 [94960.00109] controller-0 mtcAgent hbs nodeClass.cpp (4324) service_netlink_events : Warn : Management link eno2 is up 2019-03-22T16:57:20.261 [94960.00110] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down I suggest you stabilize your management network between the 2 controllers. You can always look for Error or Warn logs in mtcAgent.log as indication of unexpected conditions. The logs are kept quite short and try to indicate the issue at hand. In this case eno2 is bouncing. Eric. From: 오재욱 [mailto:kyle.oh95@gmail.com] Sent: Tuesday, March 26, 2019 10:03 PM To: MacDonald, Eric Cc: Hu, Yong; Sun, Austin; starlingx-discuss Subject: Re: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" Importance: High Hello Eric, thanks for advice. I commented my answer to your advices, 0. Please check your network and maybe specifically its support for multicast messaging. -> Could you tell me how can I check that my network supports multicast messaging? Do I need function like 'omping'? 1. If you lock controller-1 can you ping it from the active controller. -> Yes, I can ping to controller-1 from the active one, with controller-1's management ip (192.168.204.4), and controller-1's oam ip (10.10.10.4). 2. If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files -> I attached those files :) 3. You can always run ‘collect’ or ‘collect all’ on the active controller to get a summary of the system state and use those logs for off system debug of your issue. -> Thanks, I totally didn't know about those commands! 2019년 3월 26일 (화) 오후 10:45, MacDonald, Eric <Eric.MacDonald@windriver.com<mailto:Eric.MacDonald@windriver.com>>님이 작성: That alarm is indication that Maintenance is failing Heartbeat over the Management network. Please check your network and maybe specifically its support for multicast messaging. If you lock controller-1 can you ping it from the active controller. If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files You can always run ‘collect’ or ‘collect all’ on the active controller to get a summary of the system state and use those logs for off system debug of your issue. Eric MacDonald.
Thank you very much Eric, I also saw those lines, and I agree with that point. Unfortunately, I'm not sure what to do to stabilize my management network. Is that mean check physical NIC? Or, is there any command that makes network stable? Thank you again, BR, Jaewook. 2019년 3월 27일 (수) 오후 9:38, MacDonald, Eric <Eric.MacDonald@windriver.com>님이 작성:
There appears to be something wrong with your multi-cast setup as the hbsAgent is not able to setup its pulse messaging socket for the management interface.
hbsAgent.log – This log seems to have rotated due to successive socket resetup retries.
2019-03-27T10:33:55.984 [4194.3824110] controller-0 hbsAgent --- msgClass.cpp (1033) msgClassTx : Info : Creating controller-0 socket on port 2106 with address: 239.1.1.2
2019-03-27T10:33:55.984 [4194.3824111] controller-0 hbsAgent --- msgClass.cpp (1098) initSocket :Error : Failed to bind socket to address (22:Invalid argument)
2019-03-27T10:33:55.984 [4194.3824112] controller-0 hbsAgent hbs hbsAgent.cpp ( 575) _setup_pulse_messaging :Error : Cannot open multicast transmit socket - rc:34 (22:Invalid argument)
mtcAgent.log shows that your management interface is bouncing which is causing command failures.
2019-03-22T16:57:19.449 [94960.00107] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down
2019-03-22T16:57:19.449 [94960.00108] controller-0 mtcAgent hbs nodeClass.cpp (4301) service_netlink_events : Warn : eno2 is down (oper:down)
2019-03-22T16:57:19.881 [94960.00109] controller-0 mtcAgent hbs nodeClass.cpp (4324) service_netlink_events : Warn : Management link eno2 is up
2019-03-22T16:57:20.261 [94960.00110] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down
*I suggest you stabilize your management network between the 2 controllers.*
You can always look for Error or Warn logs in mtcAgent.log as indication of unexpected conditions.
The logs are kept quite short and try to indicate the issue at hand. In this case eno2 is bouncing.
Eric.
*From:* 오재욱 [mailto:kyle.oh95@gmail.com] *Sent:* Tuesday, March 26, 2019 10:03 PM *To:* MacDonald, Eric *Cc:* Hu, Yong; Sun, Austin; starlingx-discuss *Subject:* Re: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'" *Importance:* High
Hello Eric,
thanks for advice.
I commented my answer to your advices,
0. Please check your network and maybe specifically its support for multicast messaging.
-> Could you tell me how can I check that my network supports multicast messaging? Do I need function like 'omping'?
1. If you lock controller-1 can you ping it from the active controller.
-> Yes, I can ping to controller-1 from the active one, with controller-1's management ip (192.168.204.4), and controller-1's oam ip (10.10.10.4).
2. If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files
-> I attached those files :)
3. You can always run ‘collect’ or ‘collect all’ on the active controller to get a summary of the system state and use those logs for off system debug of your issue.
-> Thanks, I totally didn't know about those commands!
2019년 3월 26일 (화) 오후 10:45, MacDonald, Eric <Eric.MacDonald@windriver.com> 님이 작성:
That alarm is indication that Maintenance is failing Heartbeat over the Management network.
Please check your network and maybe specifically its support for multicast messaging.
If you lock controller-1 can you ping it from the active controller.
If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files
You can always run ‘collect’ or ‘collect all’ on the active controller to get a summary of the system state and use those logs for off system debug of your issue.
Eric MacDonald.
-- ================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea
Hello all, I finally added controller-1 successfully. At first I used 1G NIC for management network, but did not work. Today I changed 10G NIC for the network, and it worked well! Thank you very much :) Best Regards, Jaewook. 2019년 3월 27일 (수) 오후 9:38, MacDonald, Eric <Eric.MacDonald@windriver.com>님이 작성:
There appears to be something wrong with your multi-cast setup as the hbsAgent is not able to setup its pulse messaging socket for the management interface.
hbsAgent.log – This log seems to have rotated due to successive socket resetup retries.
2019-03-27T10:33:55.984 [4194.3824110] controller-0 hbsAgent --- msgClass.cpp (1033) msgClassTx : Info : Creating controller-0 socket on port 2106 with address: 239.1.1.2
2019-03-27T10:33:55.984 [4194.3824111] controller-0 hbsAgent --- msgClass.cpp (1098) initSocket :Error : Failed to bind socket to address (22:Invalid argument)
2019-03-27T10:33:55.984 [4194.3824112] controller-0 hbsAgent hbs hbsAgent.cpp ( 575) _setup_pulse_messaging :Error : Cannot open multicast transmit socket - rc:34 (22:Invalid argument)
mtcAgent.log shows that your management interface is bouncing which is causing command failures.
2019-03-22T16:57:19.449 [94960.00107] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down
2019-03-22T16:57:19.449 [94960.00108] controller-0 mtcAgent hbs nodeClass.cpp (4301) service_netlink_events : Warn : eno2 is down (oper:down)
2019-03-22T16:57:19.881 [94960.00109] controller-0 mtcAgent hbs nodeClass.cpp (4324) service_netlink_events : Warn : Management link eno2 is up
2019-03-22T16:57:20.261 [94960.00110] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down
*I suggest you stabilize your management network between the 2 controllers.*
You can always look for Error or Warn logs in mtcAgent.log as indication of unexpected conditions.
The logs are kept quite short and try to indicate the issue at hand. In this case eno2 is bouncing.
Eric.
================================================ *Jaewook Oh* (오재욱) IISTRC - Internet Infra System Technology Research Center 369 Sangdo-ro, Dongjak-gu, 06978, Seoul, Republic of Korea
participants (4)
-
Hu, Yong
-
MacDonald, Eric
-
Sun, Austin
-
오재욱