[Starlingx-discuss] rabbitmq dead bindings

[Richard, Joseph]

I'm looking at an issue (https://bugs.launchpad.net/starlingx/+bug/1835807) where bindings in rabbit (e.g. neutron q-l3-plugin) intermittently stop working during host failover (via reboot -f).  Once this happens, any RPCs on the topic where the binding has died start timing out and being dropped, which renders the stx-openstack application nearly unusable once this happens.  This can be recovered by deleting the queues those topics are bound to, or by restarting rabbit.

It looks like messages get into rabbit, which still has the bindings(seen by running rabbitmqctl -p neutron list_bindings), but rabbit ignores the bindings and treats the message as unroutable.  When I set the alternative exchange for that exchange (e.g. neutron) to the fanout for that topic, the RPCs start going through again.

I am testing turning off ha mode in rabbit to see if that stops this from occurring, but would prefer a proper solution to prevent or fix this.   Has anyone seen this before, and has a better solution for this?  Does anyone have any suggestions on debugging this, given that it clears up when rabbit restarts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190820/cdf27c44/attachment.html>