Hi Joseph, I fixed this issue now!! After check port status by sudo ip -all netns exec lsof -I, it shows that Rabbitmq only bind 15672 to ipv4. I add below changes and pass the rabbitmq deployment. conf: rabbitmq: management.listener.port: 15672 management.listener.ip: "::" Next step: I have to think how to submit this patch to support both ipv4 and ipv6 BTW, are you sure we passed this test before using OpenStack- train version? If not change helm chart, how can we pass it? Thanks! Zhipeng From: Liu, ZhipengS Sent: 2020年3月19日 10:07 To: Richard, Joseph <Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Hi Joseph, Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch) From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001 From: Zhipeng Liu <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Date: Thu, 19 Mar 2020 18:53:07 +0800 Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> --- rabbitmq/templates/configmap-etc.yaml | 8 +++++++- rabbitmq/templates/job-cluster-wait.yaml | 2 ++ rabbitmq/templates/statefulset.yaml | 8 ++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml index 87f25f5..18e2145 100644 --- a/rabbitmq/templates/configmap-etc.yaml +++ b/rabbitmq/templates/configmap-etc.yaml @@ -21,7 +21,8 @@ limitations under the License. {{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}} {{- end -}} -{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} +{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} + --- apiVersion: v1 @@ -33,4 +34,9 @@ data: {{ tuple "etc/_enabled_plugins.tpl" . | include "helm-toolkit.utils.template" | indent 4 }} rabbitmq.conf: | {{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }} + rabbitmq-env.conf: | + SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp" + CTL_ERL_ARGS="-proto_dist inet6_tcp" + erl_inetrc: | + {inet6, true}. {{ end }} diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml index bf8e710..24a30a3 100644 --- a/rabbitmq/templates/job-cluster-wait.yaml +++ b/rabbitmq/templates/job-cluster-wait.yaml @@ -66,6 +66,8 @@ spec: env: - name: RABBITMQ_ADMIN_CONNECTION value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }} + - name: RABBITMQ_CTL_ERL_ARGS + value: "-proto_dist inet6_tcp" - name: RABBIT_REPLICA_COUNT value: {{ $envAll.Values.pod.replicas.server | quote }} command: diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml index 41dc198..64fc1cf 100644 --- a/rabbitmq/templates/statefulset.yaml +++ b/rabbitmq/templates/statefulset.yaml @@ -228,6 +228,14 @@ spec: mountPath: /etc/rabbitmq/rabbitmq.conf subPath: rabbitmq.conf readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/rabbitmq-env.conf + subPath: rabbitmq-env.conf + readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/erl_inetrc + subPath: erl_inetrc + readOnly: true volumes: - name: pod-tmp emptyDir: {} -- 2.7.4 From: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>> Sent: 2020年3月19日 0:35 To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Have you tried changing envAll.Values.conf.rabbitmq.listeners.tcp? I'd need to see the changes you've made to get rabbitmq running to provide any other ideas. ________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: Wednesday, March 18, 2020 10:53 AM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Hi Joseph, I will submit one patch for helm chart change later after fixing current issue. BTW, Seems we have no rabbitmqadmin in rabbitmq pod but rabbitmq-cluster-wait job. I have done below test and have some conclusion now. 1) Rabbtimq already started normally, and start listen on [::]:5672/15672/25672 port 2) It seems 15672 port access blocked for ipv6 (5672 is OK). In ipv4 simplex setup, same test below are all OK. Any idea? rabbitmq@osh-openstack-rabbitmq-rabbitmq-0:/usr/local$ rabbitmqctl status Status of node rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local<mailto:rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local> ... Runtime OS PID: 303 OS: Linux Uptime (seconds): 6752 RabbitMQ version: 3.8.3 Node name: rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local<mailto:rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local> Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:128] Erlang processes: 426 used, 1048576 limit Scheduler run queue: 1 Cluster heartbeat timeout (net_ticktime): 60 Plugins Enabled plugin file: /etc/rabbitmq/enabled_plugins Enabled plugins: * rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_peer_discovery_k8s * rabbitmq_peer_discovery_common * rabbitmq_management_agent * amqp_client * cowboy * cowlib Totals Connection count: 0 Queue count: 0 Virtual host count: 0 Listeners Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Interface: [::], port: 15672, protocol: http, purpose: HTTP API controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... --2020-03-18 14:37:45-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:15672... failed: Connection refused. //same as we see in the log of Rabbitmq-cluster-wait pod. controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... --2020-03-18 14:37:53-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:5672... connected. controller-0:~$ wget http://[fd03::1e6d:6abf]:15672/ --2020-03-18 14:38:56-- http://[fd03::1e6d:6abf]:15672/ Connecting to [fd03::1e6d:6abf]:15672... failed: Connection refused. controller-0:~$ wget http://[fd03::1e6d:6abf]:5672/ --2020-03-18 14:39:00-- http://[fd03::1e6d:6abf]:5672/ Connecting to [fd03::1e6d:6abf]:5672... connected. Thanks! Zhipeng From: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>> Sent: 2020年3月18日 21:48 To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Can you open a review with the change you made to fix the nxdomain issue? Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0? I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running. Can you see anything in the rabbitmq when the cluster-wait pod fails? ________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: Tuesday, March 17, 2020 9:46 PM To: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Hi Joseph, After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup. Below two pods can run up. osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m However, still have another issue. In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq, but always get error. ++ active_rabbit_nodes 2020-03-17T10:31:12.124589385Z stderr F ++ wc -w 2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash 2020-03-17T10:31:12.134427089Z stderr F ++ set +x 2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last): 2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused 2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused In the log of rabbitmq node, I can see below. 2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672 2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672 2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started. 2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s 2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management 2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent 2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch 2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common I’m still working on this issue. Any comment or proposal, thanks! Thanks! Zhipeng From: Liu, ZhipengS Sent: 2020年3月16日 16:57 To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Hi Joseph, I tried many methods, but still not fixed the issue. I found there are some related issued fixed in upstream, but not yet merged to formal release. I’m discussing with them. Do you have any comment? https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55 Thanks! Zhipeng From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Hi Joseph, Thank you so much! 😊 I will work further based on your provided clue. Best Regards, Zhipeng From: Richard, Joseph <Joseph.Richard@windriver.com<mailto:Joseph.Richard@windriver.com>> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. I see a similar issue online here: https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA [http://www.google.com/images/icons/product/groups-128.png]<https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA> epmd fails to start in IPv6 environment<https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA> Posted 7/15/19 9:43 PM, 12 messages groups.google.com I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6: SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp' CTL_ERL_ARGS='-proto_dist inet6_tcp' Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in. ________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.com<mailto:zhipengs.liu@intel.com>> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex. Hi all, For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status" https://bugs.launchpad.net/starlingx/+bug/1859641 I provided a patch which can fix the known issue in LP. However, it still failed later when starting rabbitmq pod. controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h There are below error logs 2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain) In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected? I added below line manually after the line of 127.0.0.1. However, it didn’t work☹ ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 # HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0 I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215 class platform::config::hosts inherits ::platform::config::params { # The localhost should resolve to the IPv4 loopback address only, therefore # ensure the IPv6 address is removed from configured hosts resources { 'host': purge => true } $localhost = { 'localhost' => { ip => '127.0.0.1', host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4'] }, } $merged_hosts = merge($localhost, $hosts) create_resources('host', $merged_hosts, {}) } Any one can give some help? Thanks! Zhipeng