[Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status" https://bugs.launchpad.net/starlingx/+bug/1859641 I provided a patch which can fix the known issue in LP. However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs 2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected? I added below line manually after the line of 127.0.0.1. However, it didn’t work☹ ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
I see a similar issue online here: https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA [http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA Posted 7/15/19 9:43 PM, 12 messages groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6: SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp' CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________ From: Liu, ZhipengS zhipengs.liu@intel.com Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.io starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Hi Joseph,
Thank you so much! 😊 I will work further based on your provided clue.
Best Regards, Zhipeng
From: Richard, Joseph Joseph.Richard@windriver.com Sent: 2020年3月12日 5:06 To: Liu, ZhipengS zhipengs.liu@intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here: https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA [http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA Posted 7/15/19 9:43 PM, 12 messages groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6: SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp' CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Hi Joseph,
I tried many methods, but still not fixed the issue. I found there are some related issued fixed in upstream, but not yet merged to formal release. I’m discussing with them. Do you have any comment? https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks! Zhipeng
From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' Joseph.Richard@windriver.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much! 😊 I will work further based on your provided clue.
Best Regards, Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here: https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA [http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA Posted 7/15/19 9:43 PM, 12 messages groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6: SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp' CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Hi Joseph,
After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup. Below two pods can run up. osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m
However, still have another issue. In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq, but always get error. ++ active_rabbit_nodes 2020-03-17T10:31:12.124589385Z stderr F ++ wc -w 2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash 2020-03-17T10:31:12.134427089Z stderr F ++ set +x 2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last): 2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused 2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused
In the log of rabbitmq node, I can see below.
2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672 2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672 2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started. 2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s 2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management 2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent 2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch 2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common
I’m still working on this issue. Any comment or proposal, thanks!
Thanks! Zhipeng From: Liu, ZhipengS Sent: 2020年3月16日 16:57 To: 'Richard, Joseph' Joseph.Richard@windriver.com; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I tried many methods, but still not fixed the issue. I found there are some related issued fixed in upstream, but not yet merged to formal release. I’m discussing with them. Do you have any comment? https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks! Zhipeng
From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much! 😊 I will work further based on your provided clue.
Best Regards, Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here: https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA [http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA Posted 7/15/19 9:43 PM, 12 messages groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6: SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp' CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Can you open a review with the change you made to fix the nxdomain issue?
Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0?
I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running. Can you see anything in the rabbitmq when the cluster-wait pod fails?
________________________________ From: Liu, ZhipengS zhipengs.liu@intel.com Sent: Tuesday, March 17, 2020 9:46 PM To: Richard, Joseph Joseph.Richard@windriver.com; starlingx-discuss@lists.starlingx.io starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup.
Below two pods can run up.
osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m
However, still have another issue. In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq, but always get error.
++ active_rabbit_nodes 2020-03-17T10:31:12.124589385Z stderr F ++ wc -w 2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash 2020-03-17T10:31:12.134427089Z stderr F ++ set +x 2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last): 2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused 2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused
In the log of rabbitmq node, I can see below.
2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672
2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started.
2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s
2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management
2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent
2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch
2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common
I’m still working on this issue. Any comment or proposal, thanks!
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月16日 16:57 To: 'Richard, Joseph' Joseph.Richard@windriver.com; 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I tried many methods, but still not fixed the issue.
I found there are some related issued fixed in upstream, but not yet merged to formal release.
I’m discussing with them. Do you have any comment?
https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much! 😊
I will work further based on your provided clue.
Best Regards,
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here:
https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
[http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
Posted 7/15/19 9:43 PM, 12 messages
groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6:
SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp'
CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Hi Joseph,
I will submit one patch for helm chart change later after fixing current issue. BTW, Seems we have no rabbitmqadmin in rabbitmq pod but rabbitmq-cluster-wait job.
I have done below test and have some conclusion now.
1) Rabbtimq already started normally, and start listen on [::]:5672/15672/25672 port
2) It seems 15672 port access blocked for ipv6 (5672 is OK). In ipv4 simplex setup, same test below are all OK. Any idea?
rabbitmq@osh-openstack-rabbitmq-rabbitmq-0:/usr/local$ rabbitmqctl status Status of node rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local ... Runtime
OS PID: 303 OS: Linux Uptime (seconds): 6752 RabbitMQ version: 3.8.3 Node name: rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:128] Erlang processes: 426 used, 1048576 limit Scheduler run queue: 1 Cluster heartbeat timeout (net_ticktime): 60
Plugins
Enabled plugin file: /etc/rabbitmq/enabled_plugins Enabled plugins:
* rabbitmq_management * rabbitmq_web_dispatch * rabbitmq_peer_discovery_k8s * rabbitmq_peer_discovery_common * rabbitmq_management_agent * amqp_client * cowboy * cowlib
Totals
Connection count: 0 Queue count: 0 Virtual host count: 0
Listeners
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Interface: [::], port: 15672, protocol: http, purpose: HTTP API
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... --2020-03-18 14:37:45-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:15672... failed: Connection refused. //same as we see in the log of Rabbitmq-cluster-wait pod.
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... --2020-03-18 14:37:53-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust... Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:5672... connected.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:15672/ --2020-03-18 14:38:56-- http://%5Bfd03::1e6d:6abf%5D:15672/ Connecting to [fd03::1e6d:6abf]:15672... failed: Connection refused.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:5672/ --2020-03-18 14:39:00-- http://%5Bfd03::1e6d:6abf%5D:5672/ Connecting to [fd03::1e6d:6abf]:5672... connected.
Thanks! Zhipeng From: Richard, Joseph Joseph.Richard@windriver.com Sent: 2020年3月18日 21:48 To: Liu, ZhipengS zhipengs.liu@intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Can you open a review with the change you made to fix the nxdomain issue?
Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0?
I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running. Can you see anything in the rabbitmq when the cluster-wait pod fails?
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Tuesday, March 17, 2020 9:46 PM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup.
Below two pods can run up.
osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m
However, still have another issue. In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq, but always get error.
++ active_rabbit_nodes 2020-03-17T10:31:12.124589385Z stderr F ++ wc -w 2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash 2020-03-17T10:31:12.134427089Z stderr F ++ set +x 2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last): 2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused 2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused
In the log of rabbitmq node, I can see below.
2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672
2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started.
2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s
2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management
2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent
2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch
2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common
I’m still working on this issue. Any comment or proposal, thanks!
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月16日 16:57 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I tried many methods, but still not fixed the issue.
I found there are some related issued fixed in upstream, but not yet merged to formal release.
I’m discussing with them. Do you have any comment?
https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much! 😊
I will work further based on your provided clue.
Best Regards,
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here:
https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
[http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
Posted 7/15/19 9:43 PM, 12 messages
groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6:
SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp'
CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Have you tried changing envAll.Values.conf.rabbitmq.listeners.tcp?
I'd need to see the changes you've made to get rabbitmq running to provide any other ideas.
________________________________ From: Liu, ZhipengS zhipengs.liu@intel.com Sent: Wednesday, March 18, 2020 10:53 AM To: Richard, Joseph Joseph.Richard@windriver.com; starlingx-discuss@lists.starlingx.io starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I will submit one patch for helm chart change later after fixing current issue.
BTW, Seems we have no rabbitmqadmin in rabbitmq pod but rabbitmq-cluster-wait job.
I have done below test and have some conclusion now.
1) Rabbtimq already started normally, and start listen on [::]:5672/15672/25672 port
2) It seems 15672 port access blocked for ipv6 (5672 is OK). In ipv4 simplex setup, same test below are all OK. Any idea?
rabbitmq@osh-openstack-rabbitmq-rabbitmq-0:/usr/local$ rabbitmqctl status
Status of node rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local ...
Runtime
OS PID: 303
OS: Linux
Uptime (seconds): 6752
RabbitMQ version: 3.8.3
Node name: rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local
Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:128]
Erlang processes: 426 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60
Plugins
Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_peer_discovery_k8s
* rabbitmq_peer_discovery_common
* rabbitmq_management_agent
* amqp_client
* cowboy
* cowlib
Totals
Connection count: 0
Queue count: 0
Virtual host count: 0
Listeners
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
--2020-03-18 14:37:45-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:15672... failed: Connection refused. //same as we see in the log of
Rabbitmq-cluster-wait pod.
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
--2020-03-18 14:37:53-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:5672... connected.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:15672/
--2020-03-18 14:38:56-- http://%5Bfd03::1e6d:6abf%5D:15672/
Connecting to [fd03::1e6d:6abf]:15672... failed: Connection refused.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:5672/
--2020-03-18 14:39:00-- http://%5Bfd03::1e6d:6abf%5D:5672/
Connecting to [fd03::1e6d:6abf]:5672... connected.
Thanks!
Zhipeng
From: Richard, Joseph Joseph.Richard@windriver.com Sent: 2020年3月18日 21:48 To: Liu, ZhipengS zhipengs.liu@intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Can you open a review with the change you made to fix the nxdomain issue?
Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0?
I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running. Can you see anything in the rabbitmq when the cluster-wait pod fails?
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Tuesday, March 17, 2020 9:46 PM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup.
Below two pods can run up.
osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m
However, still have another issue. In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq, but always get error.
++ active_rabbit_nodes 2020-03-17T10:31:12.124589385Z stderr F ++ wc -w 2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash 2020-03-17T10:31:12.134427089Z stderr F ++ set +x 2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last): 2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused 2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused
In the log of rabbitmq node, I can see below.
2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672
2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started.
2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s
2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management
2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent
2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch
2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common
I’m still working on this issue. Any comment or proposal, thanks!
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月16日 16:57 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I tried many methods, but still not fixed the issue.
I found there are some related issued fixed in upstream, but not yet merged to formal release.
I’m discussing with them. Do you have any comment?
https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much! 😊
I will work further based on your provided clue.
Best Regards,
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here:
https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
[http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
Posted 7/15/19 9:43 PM, 12 messages
groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6:
SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp'
CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Hi Joseph,
Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch)
From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001 From: Zhipeng Liu zhipengs.liu@intel.com Date: Thu, 19 Mar 2020 18:53:07 +0800 Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup
Signed-off-by: Zhipeng Liu zhipengs.liu@intel.com --- rabbitmq/templates/configmap-etc.yaml | 8 +++++++- rabbitmq/templates/job-cluster-wait.yaml | 2 ++ rabbitmq/templates/statefulset.yaml | 8 ++++++++ 3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml index 87f25f5..18e2145 100644 --- a/rabbitmq/templates/configmap-etc.yaml +++ b/rabbitmq/templates/configmap-etc.yaml @@ -21,7 +21,8 @@ limitations under the License. {{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}} {{- end -}} -{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} +{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} + --- apiVersion: v1 @@ -33,4 +34,9 @@ data: {{ tuple "etc/_enabled_plugins.tpl" . | include "helm-toolkit.utils.template" | indent 4 }} rabbitmq.conf: | {{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }} + rabbitmq-env.conf: | + SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp" + CTL_ERL_ARGS="-proto_dist inet6_tcp" + erl_inetrc: | + {inet6, true}. {{ end }} diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml index bf8e710..24a30a3 100644 --- a/rabbitmq/templates/job-cluster-wait.yaml +++ b/rabbitmq/templates/job-cluster-wait.yaml @@ -66,6 +66,8 @@ spec: env: - name: RABBITMQ_ADMIN_CONNECTION value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }} + - name: RABBITMQ_CTL_ERL_ARGS + value: "-proto_dist inet6_tcp" - name: RABBIT_REPLICA_COUNT value: {{ $envAll.Values.pod.replicas.server | quote }} command: diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml index 41dc198..64fc1cf 100644 --- a/rabbitmq/templates/statefulset.yaml +++ b/rabbitmq/templates/statefulset.yaml @@ -228,6 +228,14 @@ spec: mountPath: /etc/rabbitmq/rabbitmq.conf subPath: rabbitmq.conf readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/rabbitmq-env.conf + subPath: rabbitmq-env.conf + readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/erl_inetrc + subPath: erl_inetrc + readOnly: true volumes: - name: pod-tmp emptyDir: {} -- 2.7.4
From: Richard, Joseph Joseph.Richard@windriver.com Sent: 2020年3月19日 0:35 To: Liu, ZhipengS zhipengs.liu@intel.com; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Have you tried changing envAll.Values.conf.rabbitmq.listeners.tcp?
I'd need to see the changes you've made to get rabbitmq running to provide any other ideas.
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Wednesday, March 18, 2020 10:53 AM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I will submit one patch for helm chart change later after fixing current issue.
BTW, Seems we have no rabbitmqadmin in rabbitmq pod but rabbitmq-cluster-wait job.
I have done below test and have some conclusion now.
1) Rabbtimq already started normally, and start listen on [::]:5672/15672/25672 port
2) It seems 15672 port access blocked for ipv6 (5672 is OK). In ipv4 simplex setup, same test below are all OK. Any idea?
rabbitmq@osh-openstack-rabbitmq-rabbitmq-0:/usr/local$ rabbitmqctl status
Status of node rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.localmailto:rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local ...
Runtime
OS PID: 303
OS: Linux
Uptime (seconds): 6752
RabbitMQ version: 3.8.3
Node name: rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.localmailto:rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local
Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:128]
Erlang processes: 426 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60
Plugins
Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_peer_discovery_k8s
* rabbitmq_peer_discovery_common
* rabbitmq_management_agent
* amqp_client
* cowboy
* cowlib
Totals
Connection count: 0
Queue count: 0
Virtual host count: 0
Listeners
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
--2020-03-18 14:37:45-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:15672... failed: Connection refused. //same as we see in the log of
Rabbitmq-cluster-wait pod.
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
--2020-03-18 14:37:53-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:5672... connected.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:15672/
--2020-03-18 14:38:56-- http://%5Bfd03::1e6d:6abf%5D:15672/
Connecting to [fd03::1e6d:6abf]:15672... failed: Connection refused.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:5672/
--2020-03-18 14:39:00-- http://%5Bfd03::1e6d:6abf%5D:5672/
Connecting to [fd03::1e6d:6abf]:5672... connected.
Thanks!
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月18日 21:48 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Can you open a review with the change you made to fix the nxdomain issue?
Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0?
I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running. Can you see anything in the rabbitmq when the cluster-wait pod fails?
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Tuesday, March 17, 2020 9:46 PM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup.
Below two pods can run up.
osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m
However, still have another issue. In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq, but always get error.
++ active_rabbit_nodes 2020-03-17T10:31:12.124589385Z stderr F ++ wc -w 2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash 2020-03-17T10:31:12.134427089Z stderr F ++ set +x 2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last): 2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused 2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused
In the log of rabbitmq node, I can see below.
2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672
2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started.
2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s
2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management
2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent
2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch
2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common
I’m still working on this issue. Any comment or proposal, thanks!
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月16日 16:57 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I tried many methods, but still not fixed the issue.
I found there are some related issued fixed in upstream, but not yet merged to formal release.
I’m discussing with them. Do you have any comment?
https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much! 😊
I will work further based on your provided clue.
Best Regards,
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here:
https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
[http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
Posted 7/15/19 9:43 PM, 12 messages
groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6:
SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp'
CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Hi Joseph,
I fixed this issue now!! After check port status by sudo ip -all netns exec lsof -I, it shows that Rabbitmq only bind 15672 to ipv4. I add below changes and pass the rabbitmq deployment. conf: rabbitmq: management.listener.port: 15672 management.listener.ip: "::"
Next step: I have to think how to submit this patch to support both ipv4 and ipv6
BTW, are you sure we passed this test before using OpenStack- train version? If not change helm chart, how can we pass it?
Thanks! Zhipeng From: Liu, ZhipengS Sent: 2020年3月19日 10:07 To: Richard, Joseph Joseph.Richard@windriver.com; starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch)
From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001 From: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Date: Thu, 19 Mar 2020 18:53:07 +0800 Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup
Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> --- rabbitmq/templates/configmap-etc.yaml | 8 +++++++- rabbitmq/templates/job-cluster-wait.yaml | 2 ++ rabbitmq/templates/statefulset.yaml | 8 ++++++++ 3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml index 87f25f5..18e2145 100644 --- a/rabbitmq/templates/configmap-etc.yaml +++ b/rabbitmq/templates/configmap-etc.yaml @@ -21,7 +21,8 @@ limitations under the License. {{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}} {{- end -}}
-{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} +{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} +
--- apiVersion: v1 @@ -33,4 +34,9 @@ data: {{ tuple "etc/_enabled_plugins.tpl" . | include "helm-toolkit.utils.template" | indent 4 }} rabbitmq.conf: | {{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }} + rabbitmq-env.conf: | + SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp" + CTL_ERL_ARGS="-proto_dist inet6_tcp" + erl_inetrc: | + {inet6, true}. {{ end }} diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml index bf8e710..24a30a3 100644 --- a/rabbitmq/templates/job-cluster-wait.yaml +++ b/rabbitmq/templates/job-cluster-wait.yaml @@ -66,6 +66,8 @@ spec: env: - name: RABBITMQ_ADMIN_CONNECTION value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }} + - name: RABBITMQ_CTL_ERL_ARGS + value: "-proto_dist inet6_tcp" - name: RABBIT_REPLICA_COUNT value: {{ $envAll.Values.pod.replicas.server | quote }} command: diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml index 41dc198..64fc1cf 100644 --- a/rabbitmq/templates/statefulset.yaml +++ b/rabbitmq/templates/statefulset.yaml @@ -228,6 +228,14 @@ spec: mountPath: /etc/rabbitmq/rabbitmq.conf subPath: rabbitmq.conf readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/rabbitmq-env.conf + subPath: rabbitmq-env.conf + readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/erl_inetrc + subPath: erl_inetrc + readOnly: true volumes: - name: pod-tmp emptyDir: {} -- 2.7.4
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月19日 0:35 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Have you tried changing envAll.Values.conf.rabbitmq.listeners.tcp?
I'd need to see the changes you've made to get rabbitmq running to provide any other ideas.
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Wednesday, March 18, 2020 10:53 AM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I will submit one patch for helm chart change later after fixing current issue.
BTW, Seems we have no rabbitmqadmin in rabbitmq pod but rabbitmq-cluster-wait job.
I have done below test and have some conclusion now.
1) Rabbtimq already started normally, and start listen on [::]:5672/15672/25672 port
2) It seems 15672 port access blocked for ipv6 (5672 is OK). In ipv4 simplex setup, same test below are all OK. Any idea?
rabbitmq@osh-openstack-rabbitmq-rabbitmq-0:/usr/local$ rabbitmqctl status
Status of node rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.localmailto:rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local ...
Runtime
OS PID: 303
OS: Linux
Uptime (seconds): 6752
RabbitMQ version: 3.8.3
Node name: rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.localmailto:rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local
Erlang configuration: Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:128]
Erlang processes: 426 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60
Plugins
Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_peer_discovery_k8s
* rabbitmq_peer_discovery_common
* rabbitmq_management_agent
* amqp_client
* cowboy
* cowlib
Totals
Connection count: 0
Queue count: 0
Virtual host count: 0
Listeners
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
--2020-03-18 14:37:45-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:15672... failed: Connection refused. //same as we see in the log of
Rabbitmq-cluster-wait pod.
controller-0:~$ wget http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
--2020-03-18 14:37:53-- http://rabbit@osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.clust...
Resolving osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)... fd03::1e6d:6abf
Connecting to osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local (osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local)|fd03::1e6d:6abf|:5672... connected.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:15672/
--2020-03-18 14:38:56-- http://%5Bfd03::1e6d:6abf%5D:15672/
Connecting to [fd03::1e6d:6abf]:15672... failed: Connection refused.
controller-0:~$ wget http://%5Bfd03::1e6d:6abf%5D:5672/
--2020-03-18 14:39:00-- http://%5Bfd03::1e6d:6abf%5D:5672/
Connecting to [fd03::1e6d:6abf]:5672... connected.
Thanks!
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月18日 21:48 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Can you open a review with the change you made to fix the nxdomain issue?
Is rabbitmqadmin able to connect if you run it from within osh-openstack-rabbitmq-rabbitmq-0?
I see from your logs that the osh-openstack-rabbitmq-cluster-wait is from before rabbitmq was running. Can you see anything in the rabbitmq when the cluster-wait pod fails?
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Tuesday, March 17, 2020 9:46 PM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
After did some changes in rabbitmq helm chart, I have fixed nxdomain issue in my ipv6 simplex setup.
Below two pods can run up.
osh-openstack-rabbitmq-cluster-wait-9rw6p 1/1 Running 0 17m osh-openstack-rabbitmq-rabbitmq-0 1/1 Running 0 17m
However, still have another issue. In pod osh-openstack-rabbitmq-cluster-wait, it will use rabbitmqadmin to connect rabbitmq, but always get error.
++ active_rabbit_nodes 2020-03-17T10:31:12.124589385Z stderr F ++ wc -w 2020-03-17T10:31:12.134367271Z stderr F ++ rabbitmqadmin_authed list nodes -f bash 2020-03-17T10:31:12.134427089Z stderr F ++ set +x 2020-03-17T10:31:12.179073378Z stderr F Traceback (most recent call last): 2020-03-17T10:31:12.179644557Z stderr F error: [Errno 111] Connection refused 2020-03-17T10:31:12.17964969Z stderr F *** Could not connect: [Errno 111] Connection refused
In the log of rabbitmq node, I can see below.
2020-03-17T15:40:27.762196613Z stdout F 2020-03-17 15:40:27.761 [info] <0.495.0> started TCP listener on [::]:5672
2020-03-17T15:40:27.807886117Z stdout F 2020-03-17 15:40:27.807 [info] <0.552.0> Management plugin: HTTP (non-TLS) listener started on port 15672
2020-03-17T15:40:27.895551288Z stdout F 2020-03-17 15:40:27.895 [info] <0.8.0> Server startup complete; 5 plugins started.
2020-03-17T15:40:27.895555144Z stdout F * rabbitmq_peer_discovery_k8s
2020-03-17T15:40:27.895557288Z stdout F * rabbitmq_management
2020-03-17T15:40:27.895559593Z stdout F * rabbitmq_management_agent
2020-03-17T15:40:27.895561807Z stdout F * rabbitmq_web_dispatch
2020-03-17T15:40:27.895564619Z stdout F * rabbitmq_peer_discovery_common
I’m still working on this issue. Any comment or proposal, thanks!
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月16日 16:57 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I tried many methods, but still not fixed the issue.
I found there are some related issued fixed in upstream, but not yet merged to formal release.
I’m discussing with them. Do you have any comment?
https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s/issues/55
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月12日 9:53 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Thank you so much! 😊
I will work further based on your provided clue.
Best Regards,
Zhipeng
From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月12日 5:06 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see a similar issue online here:
https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
[http://www.google.com/images/icons/product/groups-128.png]https://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
epmd fails to start in IPv6 environmenthttps://groups.google.com/forum/#!topic/rabbitmq-users/9ir1WK-nYBA
Posted 7/15/19 9:43 PM, 12 messages
groups.google.com
I expect that you are going to have to update rabbmitmq docker images to 3.7.16(currently 3.7.13), and then modify the helm plugin to pass in these values when using ipv6:
SERVER_ADDITIONAL_ERL_ARGS='-proto_dist inet6_tcp'
CTL_ERL_ARGS='-proto_dist inet6_tcp'
Not sure if this is possible right now, or if you will need to modify openstack-helm-infra/rabbitmq in order to be able to pass these environment variables in.
________________________________
From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, February 27, 2020 8:00 AM To: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi all,
For below LP IPv6: stx-openstack app apply failed by time out waiting for the condition "grpc_status"
https://bugs.launchpad.net/starlingx/+bug/1859641
I provided a patch which can fix the known issue in LP.
However, it still failed later when starting rabbitmq pod.
controller-0:~$ kubectl -n openstack get po NAME READY STATUS RESTARTS AGE ingress-c74489c96-jsb7n 1/1 Running 0 17h ingress-error-pages-5889bfcf6c-h8kl6 1/1 Running 0 17h mariadb-ingress-b5cd65f4b-sq46c 1/1 Running 0 17h mariadb-ingress-error-pages-6d8b79b74f-9zmpc 1/1 Running 0 17h mariadb-server-0 1/1 Running 0 17h osh-openstack-memcached-memcached-545668bdbd-9jkv2 1/1 Running 0 17h osh-openstack-rabbitmq-cluster-wait-wbnwp 0/1 Init:0/2 0 17h osh-openstack-rabbitmq-rabbitmq-0 0/1 CrashLoopBackOff 207 17h
There are below error logs
2020-02-26T07:43:36.451103975Z stderr F ++ echo osh-openstack-rabbitmq-rabbitmq-0 2020-02-26T07:43:36.45142814Z stderr F ++ awk -F - '{print $NF}' 2020-02-26T07:43:36.454160452Z stderr F + POD_INCREMENT=0 2020-02-26T07:43:36.454757022Z stderr F + '[' 0 -eq 0 ']' 2020-02-26T07:43:36.454795721Z stderr F + exec rabbitmq-server 2020-02-26T07:43:36.849021551Z stdout F ERROR: epmd error for host osh-openstack-rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local: nxdomain (non-existing domain)
In /etc/hosts file below, I just found localhost4, no localhost6. Is it expected?
I added below line manually after the line of 127.0.0.1. However, it didn’t work☹
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
# HEADER: This file was autogenerated at 2020-02-26 09:00:35 +0000 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 fd01::1 controller 169.254.202.1 pxecontroller fd00::3 oamcontroller fd01::4 controller-platform-nfs fd01::3 controller-1 fd01::2 controller-0 fd02::3 controller-1-cluster-host fd02::2 controller-0-cluster-host fd01::e8b6:66fe storage-0
I saw that in ./stx-puppet/puppet-manifests/src/modules/platform/manifests/config.pp:215
class platform::config::hosts
inherits ::platform::config::params {
# The localhost should resolve to the IPv4 loopback address only, therefore
# ensure the IPv6 address is removed from configured hosts
resources { 'host': purge => true }
$localhost = {
'localhost' => {
ip => '127.0.0.1',
host_aliases => ['localhost.localdomain', 'localhost4', 'localhost4.localdomain4']
},
}
$merged_hosts = merge($localhost, $hosts)
create_resources('host', $merged_hosts, {})
}
Any one can give some help?
Thanks! Zhipeng
Hi Joseph,
I fixed this issue now!! After check port status by sudo ip -all netns exec lsof -I, it shows that Rabbitmq only bind 15672 to ipv4. I add below changes and pass the rabbitmq deployment. conf: rabbitmq: management.listener.port: 15672 management.listener.ip: "::"
Next step: I have to think how to submit this patch to support both ipv4 and ipv6
BTW, are you sure we passed this test before using OpenStack- train version? If not change helm chart, how can we pass it?
Thanks! Zhipeng From: Liu, ZhipengS Sent: 2020年3月19日 10:07 To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch)
From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001 From: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Date: Thu, 19 Mar 2020 18:53:07 +0800 Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup
Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> --- rabbitmq/templates/configmap-etc.yaml | 8 +++++++- rabbitmq/templates/job-cluster-wait.yaml | 2 ++ rabbitmq/templates/statefulset.yaml | 8 ++++++++ 3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml index 87f25f5..18e2145 100644 --- a/rabbitmq/templates/configmap-etc.yaml +++ b/rabbitmq/templates/configmap-etc.yaml @@ -21,7 +21,8 @@ limitations under the License. {{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}} {{- end -}}
-{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} +{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}} +
--- apiVersion: v1 @@ -33,4 +34,9 @@ data: {{ tuple "etc/_enabled_plugins.tpl" . | include "helm-toolkit.utils.template" | indent 4 }} rabbitmq.conf: | {{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }} + rabbitmq-env.conf: | + SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp" + CTL_ERL_ARGS="-proto_dist inet6_tcp" + erl_inetrc: | + {inet6, true}. {{ end }} diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml index bf8e710..24a30a3 100644 --- a/rabbitmq/templates/job-cluster-wait.yaml +++ b/rabbitmq/templates/job-cluster-wait.yaml @@ -66,6 +66,8 @@ spec: env: - name: RABBITMQ_ADMIN_CONNECTION value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }} + - name: RABBITMQ_CTL_ERL_ARGS + value: "-proto_dist inet6_tcp" - name: RABBIT_REPLICA_COUNT value: {{ $envAll.Values.pod.replicas.server | quote }} command: diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml index 41dc198..64fc1cf 100644 --- a/rabbitmq/templates/statefulset.yaml +++ b/rabbitmq/templates/statefulset.yaml @@ -228,6 +228,14 @@ spec: mountPath: /etc/rabbitmq/rabbitmq.conf subPath: rabbitmq.conf readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/rabbitmq-env.conf + subPath: rabbitmq-env.conf + readOnly: true + - name: rabbitmq-etc + mountPath: /etc/rabbitmq/erl_inetrc + subPath: erl_inetrc + readOnly: true volumes: - name: pod-tmp emptyDir: {} -- 2.7.4
Great! We definitely did support this before openstack-helm and openstack-helm-infra were upversioned around December.
You should be able to control this in the rabbitmq helm plugin, so you can modify the behaviour based on ip version. https://github.com/starlingx/config/blob/master/sysinv/sysinv/sysinv/sysinv/...
Once you get passed this point, does everything else come up as well? Does the app successfully apply?
________________________________ From: Liu, ZhipengS zhipengs.liu@intel.com Sent: Thursday, March 19, 2020 5:21 AM To: Richard, Joseph Joseph.Richard@windriver.com Cc: starlingx-discuss@lists.starlingx.io starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I fixed this issue now!!
After check port status by sudo ip -all netns exec lsof -I, it shows that Rabbitmq only bind 15672 to ipv4.
I add below changes and pass the rabbitmq deployment.
conf:
rabbitmq:
management.listener.port: 15672
management.listener.ip: "::"
Next step: I have to think how to submit this patch to support both ipv4 and ipv6
BTW, are you sure we passed this test before using OpenStack- train version?
If not change helm chart, how can we pass it?
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月19日 10:07 To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch)
From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001
From: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Date: Thu, 19 Mar 2020 18:53:07 +0800
Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup
Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
---
rabbitmq/templates/configmap-etc.yaml | 8 +++++++-
rabbitmq/templates/job-cluster-wait.yaml | 2 ++
rabbitmq/templates/statefulset.yaml | 8 ++++++++
3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml
index 87f25f5..18e2145 100644
--- a/rabbitmq/templates/configmap-etc.yaml
+++ b/rabbitmq/templates/configmap-etc.yaml
@@ -21,7 +21,8 @@ limitations under the License.
{{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}}
{{- end -}}
-{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}
+{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}
+
---
apiVersion: v1
@@ -33,4 +34,9 @@ data:
{{ tuple "etc/_enabled_plugins.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
rabbitmq.conf: |
{{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }}
+ rabbitmq-env.conf: |
+ SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
+ CTL_ERL_ARGS="-proto_dist inet6_tcp"
+ erl_inetrc: |
+ {inet6, true}.
{{ end }}
diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml
index bf8e710..24a30a3 100644
--- a/rabbitmq/templates/job-cluster-wait.yaml
+++ b/rabbitmq/templates/job-cluster-wait.yaml
@@ -66,6 +66,8 @@ spec:
env:
- name: RABBITMQ_ADMIN_CONNECTION
value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }}
+ - name: RABBITMQ_CTL_ERL_ARGS
+ value: "-proto_dist inet6_tcp"
- name: RABBIT_REPLICA_COUNT
value: {{ $envAll.Values.pod.replicas.server | quote }}
command:
diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml
index 41dc198..64fc1cf 100644
--- a/rabbitmq/templates/statefulset.yaml
+++ b/rabbitmq/templates/statefulset.yaml
@@ -228,6 +228,14 @@ spec:
mountPath: /etc/rabbitmq/rabbitmq.conf
subPath: rabbitmq.conf
readOnly: true
+ - name: rabbitmq-etc
+ mountPath: /etc/rabbitmq/rabbitmq-env.conf
+ subPath: rabbitmq-env.conf
+ readOnly: true
+ - name: rabbitmq-etc
+ mountPath: /etc/rabbitmq/erl_inetrc
+ subPath: erl_inetrc
+ readOnly: true
volumes:
- name: pod-tmp
emptyDir: {}
--
2.7.4
Thanks Joseph! I guess it can go further. But keystone-db failed, which is the same as I met in my another ipv4 setup. I will build an EB based on a latest green daily build. The patch will come soon for your review.
Thanks! Zhipeng From: Richard, Joseph Joseph.Richard@windriver.com Sent: 2020年3月19日 21:20 To: Liu, ZhipengS zhipengs.liu@intel.com Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Great! We definitely did support this before openstack-helm and openstack-helm-infra were upversioned around December.
You should be able to control this in the rabbitmq helm plugin, so you can modify the behaviour based on ip version. https://github.com/starlingx/config/blob/master/sysinv/sysinv/sysinv/sysinv/...
Once you get passed this point, does everything else come up as well? Does the app successfully apply?
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, March 19, 2020 5:21 AM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Cc: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I fixed this issue now!!
After check port status by sudo ip -all netns exec lsof -I, it shows that Rabbitmq only bind 15672 to ipv4.
I add below changes and pass the rabbitmq deployment.
conf:
rabbitmq:
management.listener.port: 15672
management.listener.ip: "::"
Next step: I have to think how to submit this patch to support both ipv4 and ipv6
BTW, are you sure we passed this test before using OpenStack- train version?
If not change helm chart, how can we pass it?
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月19日 10:07 To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch)
From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001
From: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Date: Thu, 19 Mar 2020 18:53:07 +0800
Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup
Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
---
rabbitmq/templates/configmap-etc.yaml | 8 +++++++-
rabbitmq/templates/job-cluster-wait.yaml | 2 ++
rabbitmq/templates/statefulset.yaml | 8 ++++++++
3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml
index 87f25f5..18e2145 100644
--- a/rabbitmq/templates/configmap-etc.yaml
+++ b/rabbitmq/templates/configmap-etc.yaml
@@ -21,7 +21,8 @@ limitations under the License.
{{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}}
{{- end -}}
-{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}
+{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}
+
---
apiVersion: v1
@@ -33,4 +34,9 @@ data:
{{ tuple "etc/_enabled_plugins.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
rabbitmq.conf: |
{{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }}
+ rabbitmq-env.conf: |
+ SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
+ CTL_ERL_ARGS="-proto_dist inet6_tcp"
+ erl_inetrc: |
+ {inet6, true}.
{{ end }}
diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml
index bf8e710..24a30a3 100644
--- a/rabbitmq/templates/job-cluster-wait.yaml
+++ b/rabbitmq/templates/job-cluster-wait.yaml
@@ -66,6 +66,8 @@ spec:
env:
- name: RABBITMQ_ADMIN_CONNECTION
value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }}
+ - name: RABBITMQ_CTL_ERL_ARGS
+ value: "-proto_dist inet6_tcp"
- name: RABBIT_REPLICA_COUNT
value: {{ $envAll.Values.pod.replicas.server | quote }}
command:
diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml
index 41dc198..64fc1cf 100644
--- a/rabbitmq/templates/statefulset.yaml
+++ b/rabbitmq/templates/statefulset.yaml
@@ -228,6 +228,14 @@ spec:
mountPath: /etc/rabbitmq/rabbitmq.conf
subPath: rabbitmq.conf
readOnly: true
+ - name: rabbitmq-etc
+ mountPath: /etc/rabbitmq/rabbitmq-env.conf
+ subPath: rabbitmq-env.conf
+ readOnly: true
+ - name: rabbitmq-etc
+ mountPath: /etc/rabbitmq/erl_inetrc
+ subPath: erl_inetrc
+ readOnly: true
volumes:
- name: pod-tmp
emptyDir: {}
--
2.7.4
Hi Joseph,
I have updated my 2 patches, both tested on ipv4 and ipv6 setup. https://review.opendev.org/#/c/714034/ https://review.opendev.org/#/c/714038/ But still failed when deploy keystone, glance, cinder later. Common failure causes are found that need to bind host to :: instead of 0.0.0.0. Otherwise, it will only bind port to ipv4 address, which causes db connect failure or pod probe failure.
After below changes, it can run until deploying cinder. 1)Change mariadb value.yaml conf: bind adress from 0.0.0.0 to :: 2) Change keystone value.yaml wsgi_keystone: | {{- $portInt := tuple "identity" "internal" "api" $ | include "helm-toolkit.endpoints.endpoint_port_lookup" }} Listen 0.0.0.0:{{ $portInt }} change to Listen :::{{ $portInt }} 3) Change glance value.yaml Conf.Default: add bind_host = "::"
I have to check these services deploy one by one.
Thanks! Zhipeng
From: Liu, ZhipengS Sent: 2020年3月19日 22:00 To: 'Richard, Joseph' Joseph.Richard@windriver.com Cc: starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Thanks Joseph! I guess it can go further. But keystone-db failed, which is the same as I met in my another ipv4 setup. I will build an EB based on a latest green daily build. The patch will come soon for your review.
Thanks! Zhipeng From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月19日 21:20 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Cc: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Great! We definitely did support this before openstack-helm and openstack-helm-infra were upversioned around December.
You should be able to control this in the rabbitmq helm plugin, so you can modify the behaviour based on ip version. https://github.com/starlingx/config/blob/master/sysinv/sysinv/sysinv/sysinv/...
Once you get passed this point, does everything else come up as well? Does the app successfully apply?
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Thursday, March 19, 2020 5:21 AM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Cc: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I fixed this issue now!!
After check port status by sudo ip -all netns exec lsof -I, it shows that Rabbitmq only bind 15672 to ipv4.
I add below changes and pass the rabbitmq deployment.
conf:
rabbitmq:
management.listener.port: 15672
management.listener.ip: "::"
Next step: I have to think how to submit this patch to support both ipv4 and ipv6
BTW, are you sure we passed this test before using OpenStack- train version?
If not change helm chart, how can we pass it?
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月19日 10:07 To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com>; starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
Below is my patch to make rabbitmq pod start in ipv6 setup.(This is not a formal patch)
From 99fc5aff8d70645c3130552789ea51d82cd287b2 Mon Sep 17 00:00:00 2001
From: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
Date: Thu, 19 Mar 2020 18:53:07 +0800
Subject: [PATCH] Fix rabbtimq pod not starting issue in IPv6 setup
Signed-off-by: Zhipeng Liu <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com>
---
rabbitmq/templates/configmap-etc.yaml | 8 +++++++-
rabbitmq/templates/job-cluster-wait.yaml | 2 ++
rabbitmq/templates/statefulset.yaml | 8 ++++++++
3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/rabbitmq/templates/configmap-etc.yaml b/rabbitmq/templates/configmap-etc.yaml
index 87f25f5..18e2145 100644
--- a/rabbitmq/templates/configmap-etc.yaml
+++ b/rabbitmq/templates/configmap-etc.yaml
@@ -21,7 +21,8 @@ limitations under the License.
{{- $_ := print "kubernetes.default.svc." $envAll.Values.endpoints.cluster_domain_suffix | set $envAll.Values.conf.rabbitmq.cluster_formation.k8s "host" -}}
{{- end -}}
-{{- $_ := print "0.0.0.0:" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}
+{{- $_ := print ":::" ( tuple "oslo_messaging" "internal" "amqp" . | include "helm-toolkit.endpoints.endpoint_port_lookup") | set $envAll.Values.conf.rabbitmq.listeners.tcp "1" -}}
+
---
apiVersion: v1
@@ -33,4 +34,9 @@ data:
{{ tuple "etc/_enabled_plugins.tpl" . | include "helm-toolkit.utils.template" | indent 4 }}
rabbitmq.conf: |
{{ include "rabbitmq.utils.to_rabbit_config" $envAll.Values.conf.rabbitmq | indent 4 }}
+ rabbitmq-env.conf: |
+ SERVER_ADDITIONAL_ERL_ARGS="+A 128 -kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp"
+ CTL_ERL_ARGS="-proto_dist inet6_tcp"
+ erl_inetrc: |
+ {inet6, true}.
{{ end }}
diff --git a/rabbitmq/templates/job-cluster-wait.yaml b/rabbitmq/templates/job-cluster-wait.yaml
index bf8e710..24a30a3 100644
--- a/rabbitmq/templates/job-cluster-wait.yaml
+++ b/rabbitmq/templates/job-cluster-wait.yaml
@@ -66,6 +66,8 @@ spec:
env:
- name: RABBITMQ_ADMIN_CONNECTION
value: {{ tuple "oslo_messaging" "internal" "user" "http" $envAll | include "helm-toolkit.endpoints.authenticated_endpoint_uri_lookup" | quote }}
+ - name: RABBITMQ_CTL_ERL_ARGS
+ value: "-proto_dist inet6_tcp"
- name: RABBIT_REPLICA_COUNT
value: {{ $envAll.Values.pod.replicas.server | quote }}
command:
diff --git a/rabbitmq/templates/statefulset.yaml b/rabbitmq/templates/statefulset.yaml
index 41dc198..64fc1cf 100644
--- a/rabbitmq/templates/statefulset.yaml
+++ b/rabbitmq/templates/statefulset.yaml
@@ -228,6 +228,14 @@ spec:
mountPath: /etc/rabbitmq/rabbitmq.conf
subPath: rabbitmq.conf
readOnly: true
+ - name: rabbitmq-etc
+ mountPath: /etc/rabbitmq/rabbitmq-env.conf
+ subPath: rabbitmq-env.conf
+ readOnly: true
+ - name: rabbitmq-etc
+ mountPath: /etc/rabbitmq/erl_inetrc
+ subPath: erl_inetrc
+ readOnly: true
volumes:
- name: pod-tmp
emptyDir: {}
--
2.7.4
Hi Josesh,
I have added 2 more patches for your review. https://review.opendev.org/714894 https://review.opendev.org/714898
So far, it can deploy services until heat. But I still have an intermittent issue that ***-ks-user jobs always need restart several times to get task finished. Sometimes, it may cause endpoint job failure and block deployment procedure. Log looks like below, still not find the root cause. Could you help on this point? Thanks!
Failed to discover available identity versions when contacting http://keystone-api.openstack.svc.cluster.local:5000/v3. Attempting to parse version from URL. Unable to establish connection to http://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens: HTTPConnectionPool(host='keystone-api.openstack.svc.cluster.local', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f857a551250>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
Thanks! Zhipeng From: Liu, ZhipengS Sent: 2020年3月23日 21:50 To: 'Richard, Joseph' Joseph.Richard@windriver.com Cc: 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I have updated my 2 patches, both tested on ipv4 and ipv6 setup. https://review.opendev.org/#/c/714034/ https://review.opendev.org/#/c/714038/ But still failed when deploy keystone, glance, cinder later. Common failure causes are found that need to bind host to :: instead of 0.0.0.0. Otherwise, it will only bind port to ipv4 address, which causes db connect failure or pod probe failure.
After below changes, it can run until deploying cinder. 1)Change mariadb value.yaml conf: bind adress from 0.0.0.0 to :: 2) Change keystone value.yaml wsgi_keystone: | {{- $portInt := tuple "identity" "internal" "api" $ | include "helm-toolkit.endpoints.endpoint_port_lookup" }} Listen 0.0.0.0:{{ $portInt }} change to Listen :::{{ $portInt }} 3) Change glance value.yaml Conf.Default: add bind_host = "::"
I have to check these services deploy one by one.
Thanks! Zhipeng
I see at least one place [1] where keystone is configured to listen on 0.0.0.0, so that's probably why.
[1] https://github.com/openstack/openstack-helm/blob/master/keystone/values.yaml...
________________________________ From: Liu, ZhipengS zhipengs.liu@intel.com Sent: Wednesday, March 25, 2020 5:19 AM To: Richard, Joseph Joseph.Richard@windriver.com Cc: starlingx-discuss@lists.starlingx.io starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Josesh,
I have added 2 more patches for your review.
https://review.opendev.org/714894
https://review.opendev.org/714898
So far, it can deploy services until heat. But I still have an intermittent issue that
***-ks-user jobs always need restart several times to get task finished.
Sometimes, it may cause endpoint job failure and block deployment procedure.
Log looks like below, still not find the root cause. Could you help on this point? Thanks!
Failed to discover available identity versions when contacting http://keystone-api.openstack.svc.cluster.local:5000/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens: HTTPConnectionPool(host='keystone-api.openstack.svc.cluster.local', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f857a551250>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月23日 21:50 To: 'Richard, Joseph' Joseph.Richard@windriver.com Cc: 'starlingx-discuss@lists.starlingx.io' starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I have updated my 2 patches, both tested on ipv4 and ipv6 setup.
https://review.opendev.org/#/c/714034/
https://review.opendev.org/#/c/714038/
But still failed when deploy keystone, glance, cinder later.
Common failure causes are found that need to bind host to :: instead of 0.0.0.0.
Otherwise, it will only bind port to ipv4 address, which causes db connect failure or pod probe failure.
After below changes, it can run until deploying cinder.
1)Change mariadb value.yaml
conf: bind adress from 0.0.0.0 to ::
2) Change keystone value.yaml
wsgi_keystone: |
{{- $portInt := tuple "identity" "internal" "api" $ | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
Listen 0.0.0.0:{{ $portInt }} change to Listen :::{{ $portInt }}
3) Change glance value.yaml
Conf.Default:
add bind_host = "::"
I have to check these services deploy one by one.
Thanks!
Zhipeng
But this place has already been changed in my patch
https://review.opendev.org/714894
Zhipeng From: Richard, Joseph Joseph.Richard@windriver.com Sent: 2020年3月25日 22:08 To: Liu, ZhipengS zhipengs.liu@intel.com Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see at least one place [1] where keystone is configured to listen on 0.0.0.0, so that's probably why.
[1] https://github.com/openstack/openstack-helm/blob/master/keystone/values.yaml...
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Wednesday, March 25, 2020 5:19 AM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Cc: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Josesh,
I have added 2 more patches for your review.
https://review.opendev.org/714894
https://review.opendev.org/714898
So far, it can deploy services until heat. But I still have an intermittent issue that
***-ks-user jobs always need restart several times to get task finished.
Sometimes, it may cause endpoint job failure and block deployment procedure.
Log looks like below, still not find the root cause. Could you help on this point? Thanks!
Failed to discover available identity versions when contacting http://keystone-api.openstack.svc.cluster.local:5000/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens: HTTPConnectionPool(host='keystone-api.openstack.svc.cluster.local', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f857a551250>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月23日 21:50 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I have updated my 2 patches, both tested on ipv4 and ipv6 setup.
https://review.opendev.org/#/c/714034/
https://review.opendev.org/#/c/714038/
But still failed when deploy keystone, glance, cinder later.
Common failure causes are found that need to bind host to :: instead of 0.0.0.0.
Otherwise, it will only bind port to ipv4 address, which causes db connect failure or pod probe failure.
After below changes, it can run until deploying cinder.
1)Change mariadb value.yaml
conf: bind adress from 0.0.0.0 to ::
2) Change keystone value.yaml
wsgi_keystone: |
{{- $portInt := tuple "identity" "internal" "api" $ | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
Listen 0.0.0.0:{{ $portInt }} change to Listen :::{{ $portInt }}
3) Change glance value.yaml
Conf.Default:
add bind_host = "::"
I have to check these services deploy one by one.
Thanks!
Zhipeng
Hi Joseph and all,
So far, OpenStack can be deployed successfully in my ipv6 simplex setup!
I have fixed the intermittent issue I mentioned below. Need to increase process from 1 to 5 for wsgi_keystone!
2 related patches under review, test pass on both ipv4 and ipv6 simplex setup.
https://review.opendev.org/714894
https://review.opendev.org/714898
Appreciate for your comment!
Thanks! Zhipeng From: Liu, ZhipengS zhipengs.liu@intel.com Sent: 2020年3月26日 9:39 To: Richard, Joseph Joseph.Richard@windriver.com Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
But this place has already been changed in my patch
https://review.opendev.org/714894
Zhipeng From: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Sent: 2020年3月25日 22:08 To: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Cc: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
I see at least one place [1] where keystone is configured to listen on 0.0.0.0, so that's probably why.
[1] https://github.com/openstack/openstack-helm/blob/master/keystone/values.yaml...
________________________________ From: Liu, ZhipengS <zhipengs.liu@intel.commailto:zhipengs.liu@intel.com> Sent: Wednesday, March 25, 2020 5:19 AM To: Richard, Joseph <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Cc: starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Josesh,
I have added 2 more patches for your review.
https://review.opendev.org/714894
https://review.opendev.org/714898
So far, it can deploy services until heat. But I still have an intermittent issue that
***-ks-user jobs always need restart several times to get task finished.
Sometimes, it may cause endpoint job failure and block deployment procedure.
Log looks like below, still not find the root cause. Could you help on this point? Thanks!
Failed to discover available identity versions when contacting http://keystone-api.openstack.svc.cluster.local:5000/v3. Attempting to parse version from URL.
Unable to establish connection to http://keystone-api.openstack.svc.cluster.local:5000/v3/auth/tokens: HTTPConnectionPool(host='keystone-api.openstack.svc.cluster.local', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f857a551250>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
Thanks!
Zhipeng
From: Liu, ZhipengS Sent: 2020年3月23日 21:50 To: 'Richard, Joseph' <Joseph.Richard@windriver.commailto:Joseph.Richard@windriver.com> Cc: 'starlingx-discuss@lists.starlingx.io' <starlingx-discuss@lists.starlingx.iomailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [Starlingx-discuss] IPv6 issue when apply OpenStack on IPv6 simplex.
Hi Joseph,
I have updated my 2 patches, both tested on ipv4 and ipv6 setup.
https://review.opendev.org/#/c/714034/
https://review.opendev.org/#/c/714038/
But still failed when deploy keystone, glance, cinder later.
Common failure causes are found that need to bind host to :: instead of 0.0.0.0.
Otherwise, it will only bind port to ipv4 address, which causes db connect failure or pod probe failure.
After below changes, it can run until deploying cinder.
1)Change mariadb value.yaml
conf: bind adress from 0.0.0.0 to ::
2) Change keystone value.yaml
wsgi_keystone: |
{{- $portInt := tuple "identity" "internal" "api" $ | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
Listen 0.0.0.0:{{ $portInt }} change to Listen :::{{ $portInt }}
3) Change glance value.yaml
Conf.Default:
add bind_host = "::"
I have to check these services deploy one by one.
Thanks!
Zhipeng
participants (2)
-
Liu, ZhipengS
-
Richard, Joseph