There appears to be something wrong with your multi-cast setup as the hbsAgent is not able to setup its pulse messaging socket for the management interface.
hbsAgent.log – This log seems to have rotated due to successive socket resetup retries.
2019-03-27T10:33:55.984 [4194.3824110] controller-0 hbsAgent --- msgClass.cpp (1033) msgClassTx : Info : Creating controller-0 socket on port 2106 with address: 239.1.1.2
2019-03-27T10:33:55.984 [4194.3824111] controller-0 hbsAgent --- msgClass.cpp (1098) initSocket :Error : Failed to bind socket to address (22:Invalid argument)
2019-03-27T10:33:55.984 [4194.3824112] controller-0 hbsAgent hbs hbsAgent.cpp ( 575) _setup_pulse_messaging :Error : Cannot open multicast transmit socket - rc:34 (22:Invalid argument)
mtcAgent.log shows that your management interface is bouncing which is causing command failures.
2019-03-22T16:57:19.449 [94960.00107] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down
2019-03-22T16:57:19.449 [94960.00108] controller-0 mtcAgent hbs nodeClass.cpp (4301) service_netlink_events : Warn : eno2 is down (oper:down)
2019-03-22T16:57:19.881 [94960.00109] controller-0 mtcAgent hbs nodeClass.cpp (4324) service_netlink_events : Warn : Management link eno2 is up
2019-03-22T16:57:20.261 [94960.00110] controller-0 mtcAgent hbs nodeClass.cpp (4288) service_netlink_events : Warn : Management link eno2 is down
I suggest you stabilize your management network between the 2 controllers.
You can always look for Error or Warn logs in mtcAgent.log as indication of unexpected conditions.
The logs are kept quite short and try to indicate the issue at hand. In this case eno2 is bouncing.
Eric.