[Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'"

[MacDonald, Eric]

There appears to be something wrong with your multi-cast setup as the hbsAgent is not able to setup its pulse messaging socket for the management interface.

hbsAgent.log – This log seems to have rotated due to successive socket resetup retries.
2019-03-27T10:33:55.984 [4194.3824110] controller-0 hbsAgent --- msgClass.cpp      (1033) msgClassTx              : Info : Creating controller-0 socket on port 2106 with address: 239.1.1.2
2019-03-27T10:33:55.984 [4194.3824111] controller-0 hbsAgent --- msgClass.cpp      (1098) initSocket              :Error : Failed to bind socket to address (22:Invalid argument)
2019-03-27T10:33:55.984 [4194.3824112] controller-0 hbsAgent hbs hbsAgent.cpp      ( 575) _setup_pulse_messaging  :Error : Cannot open multicast transmit socket - rc:34 (22:Invalid argument)

mtcAgent.log shows that your management interface is bouncing which is causing command failures.
2019-03-22T16:57:19.449 [94960.00107] controller-0 mtcAgent hbs nodeClass.cpp     (4288) service_netlink_events  : Warn : Management link eno2 is down
2019-03-22T16:57:19.449 [94960.00108] controller-0 mtcAgent hbs nodeClass.cpp     (4301) service_netlink_events  : Warn : eno2 is down (oper:down)
2019-03-22T16:57:19.881 [94960.00109] controller-0 mtcAgent hbs nodeClass.cpp     (4324) service_netlink_events  : Warn : Management link eno2 is up
2019-03-22T16:57:20.261 [94960.00110] controller-0 mtcAgent hbs nodeClass.cpp     (4288) service_netlink_events  : Warn : Management link eno2 is down

I suggest you stabilize your management network between the 2 controllers.

You can always look for Error or Warn logs in mtcAgent.log as indication of unexpected conditions.
The logs are kept quite short and try to indicate the issue at hand. In this case eno2 is bouncing.

Eric.

From: 오재욱 [mailto:kyle.oh95 at gmail.com]
Sent: Tuesday, March 26, 2019 10:03 PM
To: MacDonald, Eric
Cc: Hu, Yong; Sun, Austin; starlingx-discuss
Subject: Re: [Starlingx-discuss] Fwd: [starlingx-discuss] [critical error after deployment] About "persistent critical 'Management Network'"
Importance: High

Hello Eric,
thanks for advice.

I commented my answer to your advices,

0. Please check your network and maybe specifically its support for multicast messaging.
-> Could you tell me how can I check that my network supports multicast messaging? Do I need function like 'omping'?

1. If you lock controller-1 can you ping it from the active controller.
-> Yes, I can ping to controller-1 from the active one, with controller-1's management ip (192.168.204.4), and controller-1's oam ip (10.10.10.4).

2. If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files
-> I attached those files :)

3. You can always run ‘collect’ or ‘collect all’ on the active controller to get a  summary of the system state and use those logs for off system debug of your issue.
-> Thanks, I totally didn't know about those commands!

2019년 3월 26일 (화) 오후 10:45, MacDonald, Eric <Eric.MacDonald at windriver.com<mailto:Eric.MacDonald at windriver.com>>님이 작성:
That alarm is indication that Maintenance is failing Heartbeat over the Management network.

Please check your network and maybe specifically its support for multicast messaging.

If you lock controller-1 can you ping it from the active controller.

If you need further assistance please provide the active controller’s mtcAgent.log and hbsAgent.log files

You can always run ‘collect’ or ‘collect all’ on the active controller to get a  summary of the system state and use those logs for off system debug of your issue.

Eric MacDonald.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190327/3d471c28/attachment.html>