Hi Joseph,

 

I fixed this issue now!!

After check port status by sudo ip -all netns exec lsof -I, it shows that Rabbitmq only bind 15672 to ipv4.

I add below changes and pass the rabbitmq deployment.

conf:

      rabbitmq:

        management.listener.port: 15672

       management.listener.ip: "::"

 

Next step: I have to think how to submit this patch to support both ipv4 and ipv6

 

BTW, are you sure we passed this test before using OpenStack- train version?

If not change helm chart, how can we pass it?

 

Thanks!

Zhipeng

From: Liu, ZhipengS
Sent: 2020
319 10:07
To: Richard, Joseph <Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Hi Joseph,

 

Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch)

 

From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001

From: Zhipeng Liu <zhipengs.liu@intel.com>

Date: Thu, 19 Mar 2020 18:53:07 +0800

Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup

 

Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.com>

---

rabbitmq/templates/configmap-etc.yaml    | 8 +++++++-

rabbitmq/templates/job-cluster-wait.yaml | 2 ++

rabbitmq/templates/statefulset.yaml      | 8 ++++++++

3 files changed, 17 insertions(+), 1 deletion(-)

 

diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml

index 87f25f5..18e2145 100644

--- a/rabbitmq/templates/configmap-etc.yaml

+++ b/rabbitmq/templates/configmap-etc.yaml

@@ -21,7 +21,8 @@ limitations under the License.

{{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}}

{{- end -}}

 

-{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}

+{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}

+

 

 ---

apiVersion: v1

@@ -33,4 +34,9 @@ data:

{{ tuple "etc/_enabled_plugins.tpl" . | include  "helm-toolkit.utils.template" | indent 4 }}

   rabbitmq.conf: |

{{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }}

+  rabbitmq-env.conf: |

+    SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"

+    CTL_ERL_ARGS="-proto_dist inet6_tcp"

+  erl_inetrc: |

+    {inet6, true}.

{{ end }}

diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml

index bf8e710..24a30a3 100644

--- a/rabbitmq/templates/job-cluster-wait.yaml

+++ b/rabbitmq/templates/job-cluster-wait.yaml

@@ -66,6 +66,8 @@ spec:

           env:

             - name: RABBITMQ_ADMIN_CONNECTION

               value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }}

+            - name: RABBITMQ_CTL_ERL_ARGS

+              value: "-proto_dist inet6_tcp"

             - name: RABBIT_REPLICA_COUNT

               value: {{ $envAll.Values.pod.replicas.server | quote }}

           command:

diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml

index 41dc198..64fc1cf 100644

--- a/rabbitmq/templates/statefulset.yaml

+++ b/rabbitmq/templates/statefulset.yaml

@@ -228,6 +228,14 @@ spec:

               mountPath: /etc/rabbitmq/rabbitmq.conf

               subPath: rabbitmq.conf

               readOnly: true

+            - name: rabbitmq-etc

+              mountPath: /etc/rabbitmq/rabbitmq-env.conf

+              subPath: rabbitmq-env.conf

+              readOnly: true

+            - name: rabbitmq-etc

+              mountPath: /etc/rabbitmq/erl_inetrc

+              subPath: erl_inetrc

+              readOnly: true

       volumes:

         - name: pod-tmp

           emptyDir: {}

--

2.7.4

 

From: Richard, Joseph <Joseph.Richard@windriver.com>
Sent: 2020
319 0:35
To: Liu, ZhipengS <zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Have you tried changing envAll.Values.conf.rabbitmq.listeners.tcp?

 

I'd need to see the changes you've made to get rabbitmq running to provide any other ideas.

 


From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: Wednesday, March 18, 2020 10:53 AM
To: Richard, Joseph <Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Hi Joseph,

 

I will submit one patch for helm chart change later after fixing current issue.

BTW, Seems we have no rabbitmqadmin in rabbitmq pod but rabbitmq-cluster-wait job.

 

I have done below test and have some conclusion now.

1)      Rabbtimq already started normally, and start listen on [::]:5672/15672/25672 port

2)      It seems 15672 port access blocked for ipv6 (5672 is OK). In ipv4 simplex setup, same test below are all OK. Any idea?

          

 

rabbitmq@osh-openstack-rabbitmq-rabbitmq-0:/usr/local$ rabbitmqctl status

Status of node rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local ...

Runtime

 

OS PID: 303

OS: Linux

Uptime (seconds): 6752

RabbitMQ version: 3.8.3

Node name: rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local

Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:128]

Erlang processes: 426 used, 1048576 limit

Scheduler run queue: 1

Cluster heartbeat timeout (net_ticktime): 60

 

Plugins

 

Enabled plugin file: /etc/rabbitmq/enabled_plugins

Enabled plugins:

 

* rabbitmq_management

* rabbitmq_web_dispatch

* rabbitmq_peer_discovery_k8s

* rabbitmq_peer_discovery_common

* rabbitmq_management_agent

* amqp_client

* cowboy

* cowlib

 

Totals

 

Connection count: 0

Queue count: 0

Virtual host count: 0

 

Listeners

 

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication

Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Interface: [::], port: 15672, protocol: http, purpose: HTTP API

 

controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:15672/

--2020-03-18 14:37:45--  http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:15672/

Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf

Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:15672... failed: Connection refused. //same as we see in the log of

Rabbitmq-cluster-wait pod.

 

controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:5672/

--2020-03-18 14:37:53--  http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:5672/

Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf

Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:5672... connected.

 

controller-0:~$ wget http://[fd03::1e6d:6abf]:15672/

--2020-03-18 14:38:56--  http://[fd03::1e6d:6abf]:15672/

Connecting to [fd03::1e6d:6abf]:15672... failed: Connection refused.

 

controller-0:~$ wget http://[fd03::1e6d:6abf]:5672/

--2020-03-18 14:39:00--  http://[fd03::1e6d:6abf]:5672/

Connecting to [fd03::1e6d:6abf]:5672... connected.

 

Thanks!

Zhipeng

From: Richard, Joseph <Joseph.Richard@windriver.com>
Sent: 2020
318 21:48
To: Liu, ZhipengS <zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Can you open a review with the change you made to fix the nxdomain issue?

 

Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0?

 

I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running.  Can you see anything in the rabbitmq when the cluster-wait pod fails?

 

 


From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: Tuesday, March 17, 2020 9:46 PM
To: Richard, Joseph <Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Hi Joseph,

 

After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup.

Below two pods can run up.

osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m
osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m

 

However, still have another issue.
In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq,
but always get error. 

++ active_rabbit_nodes
2020-03-17T10:31:12.124589385Z stderr F ++ wc -w
2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash
2020-03-17T10:31:12.134427089Z stderr F ++ set +x
2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last):
2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused
2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused

 

In the log of rabbitmq node, I can see below.

 

2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672

2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672

2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started.

2020-03-17T15:40:27.895555144Z stdout F  * rabbitmq_peer_discovery_k8s

2020-03-17T15:40:27.895557288Z stdout F  * rabbitmq_management

2020-03-17T15:40:27.895559593Z stdout F  * rabbitmq_management_agent

2020-03-17T15:40:27.895561807Z stdout F  * rabbitmq_web_dispatch

2020-03-17T15:40:27.895564619Z stdout F  * rabbitmq_peer_discovery_common

 

I’m still working on this issue. Any comment or proposal, thanks!

 

Thanks!

Zhipeng

From: Liu, ZhipengS
Sent: 2020
316 16:57
To: 'Richard, Joseph' <Joseph.Richard@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.io>
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Hi Joseph,

 

I tried many methods, but still not fixed the issue.

I found there are some related issued fixed in upstream, but not yet merged to formal release.

I’m discussing with them. Do you have any comment?

https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55

 

Thanks!

Zhipeng

 

From: Liu, ZhipengS
Sent: 2020
312 9:53
To: 'Richard, Joseph' <Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Hi Joseph,

 

Thank you so much! 😊

I will work further based on your provided clue.

 

Best Regards,

Zhipeng

 

From: Richard, Joseph <Joseph.Richard@windriver.com>
Sent: 2020
312 5:06
To: Liu, ZhipengS <zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

I see a similar issue online here:

Posted 7/15/19 9:43 PM, 12 messages

groups.google.com

 

 

I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6:

SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp'

CTL_ERL_ARGS='-proto_dist inet6_tcp'

 

Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.

 


From: Liu, ZhipengS <zhipengs.liu@intel.com>
Sent: Thursday, February 27, 2020 8:00 AM
To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>
Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.

 

Hi all,

 

For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"

https://bugs.launchpad.net/starlingx/+bug/1859641

I provided a patch which can fix the known issue in LP.

However, it still failed later when starting rabbitmq pod.

 

controller-0:~$ kubectl -n openstack get po
NAME READY STATUS RESTARTS AGE
ingress-c74489c96-jsb7n 1/1 Running 0 17h
ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h
mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h
mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h
mariadb-server-0 1/1 Running 0 17h
osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h
osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h
osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h

 

There are below error logs

2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0
2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}'
2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0
2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']'
2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server
2020-02-26T07:43:36.849021551Z stdout
F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)

 

In /etc/hosts file below, I just found localhost4, no localhost6.  Is it expected?

I added below line manually after the line of 127.0.0.1. However, it didn’t work

::1             localhost       localhost.localdomain localhost6 localhost6.localdomain6

 

# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000
# HEADER: by puppet. While it can still be managed manually, it
# HEADER: is definitely not recommended.
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
fd01::1 controller
169.254.202.1 pxecontroller
fd00::3 oamcontroller
fd01::4 controller-platform-nfs
fd01::3 controller-1
fd01::2 controller-0
fd02::3 controller-1-cluster-host
fd02::2 controller-0-cluster-host
fd01::e8b6:66fe storage-0

 

I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215

class platform::config::hosts

  inherits ::platform::config::params {

  # The localhost should resolve to the IPv4 loopback address only, therefore

  # ensure the IPv6 address is removed from configured hosts

  resources { 'host': purge => true }

  $localhost = {

    'localhost' => {

      ip => '127.0.0.1',

      host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']

    },

  }

  $merged_hosts = merge($localhost, $hosts)

  create_resources('host', $merged_hosts, {})

}

 

Any one can give some help?

Thanks!
Zhipeng