Hi Joseph,
I will submit one patch for helm chart change later after fixing current issue.
BTW, Seems we have no rabbitmqadmin in rabbitmq pod but rabbitmq-cluster-wait job.
I have done below test and have some conclusion now.
1)
Rabbtimq already started normally, and start listen on [::]:5672/15672/25672 port
2)
It seems 15672 port access blocked for ipv6 (5672 is OK). In ipv4 simplex setup, same test below are all OK. Any idea?
rabbitmq@osh-openstack-rabbitmq-rabbitmq-0:/usr/local$
rabbitmqctl status
Status of node rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local ...
Runtime
OS PID: 303
OS: Linux
Uptime (seconds): 6752
RabbitMQ version: 3.8.3
Node name: rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local
Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:128]
Erlang processes: 426 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60
Plugins
Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_peer_discovery_k8s
* rabbitmq_peer_discovery_common
* rabbitmq_management_agent
* amqp_client
* cowboy
* cowlib
Totals
Connection count: 0
Queue count: 0
Virtual host count: 0
Listeners
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:15672/
--2020-03-18 14:37:45-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:15672/
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:15672...
failed: Connection refused. //same as we see in the log of
Rabbitmq-cluster-wait pod.
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:5672/
--2020-03-18 14:37:53-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:5672/
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:5672...
connected.
controller-0:~$ wget http://[fd03::1e6d:6abf]:15672/
--2020-03-18 14:38:56-- http://[fd03::1e6d:6abf]:15672/
Connecting to [fd03::1e6d:6abf]:15672...
failed: Connection refused.
controller-0:~$ wget http://[fd03::1e6d:6abf]:5672/
--2020-03-18 14:39:00-- http://[fd03::1e6d:6abf]:5672/
Connecting to [fd03::1e6d:6abf]:5672... connected.
Thanks!
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.com>
Sent: 2020年3月18日
21:48
To: Liu, ZhipengS <zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Can you open a review with the change you made to fix the nxdomain issue?
Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0?
I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running. Can you see anything in the rabbitmq
when the cluster-wait pod fails?
From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: Tuesday, March 17, 2020 9:46 PM
To: Richard, Joseph <Joseph.Richard@windriver.com>;
starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup.
Below two pods can run up.
osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m
osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m
However, still have another issue.
In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq,
but always get error.
++ active_rabbit_nodes
2020-03-17T10:31:12.124589385Z stderr F ++ wc -w
2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash
2020-03-17T10:31:12.134427089Z stderr F ++ set +x
2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last):
2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused
2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused
In the log of rabbitmq node, I can see below.
2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672
2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started
on port 15672
2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started.
2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s
2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management
2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent
2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch
2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common
I’m still working on this issue. Any comment or proposal, thanks!
Thanks!
Zhipeng
From: Liu, ZhipengS
Sent: 2020年3月16日
16:57
To: 'Richard, Joseph' <Joseph.Richard@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I tried many methods, but still not fixed the issue.
I found there are some related issued fixed in upstream, but not yet merged to formal release.
I’m discussing with them. Do you have any comment?
https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks!
Zhipeng
From: Liu, ZhipengS
Sent: 2020年3月12日
9:53
To: 'Richard, Joseph' <Joseph.Richard@windriver.com>;
starlingx-discuss@lists.starlingx.io
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much!
😊
I will work further based on your provided clue.
Best Regards,
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.com>
Sent: 2020年3月12日
5:06
To: Liu, ZhipengS <zhipengs.liu@intel.com>;
starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here:
|
groups.google.com |
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6:
SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp'
CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: Thursday, February 27, 2020 8:00 AM
To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>
Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po
NAME READY STATUS RESTARTS AGE
ingress-c74489c96-jsb7n 1/1 Running 0 17h
ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h
mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h
mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h
mariadb-server-0 1/1 Running 0 17h
osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h
osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h
osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0
2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}'
2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0
2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']'
2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server
2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:
nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000
# HEADER: by puppet. While it can still be managed manually, it
# HEADER: is definitely not recommended.
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
fd01::1 controller
169.254.202.1 pxecontroller
fd00::3 oamcontroller
fd01::4 controller-platform-nfs
fd01::3 controller-1
fd01::2 controller-0
fd02::3 controller-1-cluster-host
fd02::2 controller-0-cluster-host
fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks!
Zhipeng