[Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud
Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
I thought that dist-cloud sub-nodes are Kubernetes only - without OpenStack support... brucej From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Friday, April 17, 2020 5:45 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
Hi Bruce: No, for stx.3.0 the openstack on subcloud works well . central cloud does not support openstack . Thanks. BR Austin Sun. From: Jones, Bruce E <bruce.e.jones@intel.com> Sent: Friday, April 17, 2020 11:01 PM To: Sun, Austin <austin.sun@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: [starlingx] cinder bootstrap failed on subcloud I thought that dist-cloud sub-nodes are Kubernetes only - without OpenStack support... brucej From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Friday, April 17, 2020 5:45 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
Hi Andy: I deployed dist-cloud in virtual env based on 20200414, but openstack application on subcloud deployed failed .
From cinder-api po logs:
: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1 controllers This url seems override from helm keystone: host_fqdn_override: admin: host: keystone-admin.openstack.svc.cluster.subcloud1 public: {} and introduced in commit [1] by you. is these any additional config needed to bring up openstack on subcloud besides the deploy guide [2]? [1] https://review.opendev.org/#/c/697361/1 [2] https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud... Thanks. BR Austin Sun. From: Sun, Austin <austin.sun@intel.com> Sent: Saturday, April 18, 2020 9:06 AM To: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Bruce: No, for stx.3.0 the openstack on subcloud works well . central cloud does not support openstack . Thanks. BR Austin Sun. From: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Sent: Friday, April 17, 2020 11:01 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [starlingx] cinder bootstrap failed on subcloud I thought that dist-cloud sub-nodes are Kubernetes only - without OpenStack support... brucej From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Friday, April 17, 2020 5:45 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
Hi Austin, What's the output of "openstack endpoint list" on containerized openstack like? keystone-admin should be show up as identity's admin endpoint .... if not, that could be a bug. Also what's the output of "kubectl get ingress --all-namespaces" or "kubectl get ingress --openstack"? I want to see whether the ingress for keystone-admin is created or not. Andy ________________________________ From: Sun, Austin <austin.sun@intel.com> Sent: Tuesday, April 28, 2020 4:34 AM To: Ning, Antai (Andy) <Andy.Ning@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Cc: Jones, Bruce E <bruce.e.jones@intel.com> Subject: FW: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Andy: I deployed dist-cloud in virtual env based on 20200414, but openstack application on subcloud deployed failed .
From cinder-api po logs:
: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1 controllers This url seems override from helm keystone: host_fqdn_override: admin: host: keystone-admin.openstack.svc.cluster.subcloud1 public: {} and introduced in commit [1] by you. is these any additional config needed to bring up openstack on subcloud besides the deploy guide [2]? [1] https://review.opendev.org/#/c/697361/1 [2] https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud... Thanks. BR Austin Sun. From: Sun, Austin <austin.sun@intel.com> Sent: Saturday, April 18, 2020 9:06 AM To: Jones, Bruce E <bruce.e.jones@intel.com>; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Bruce: No, for stx.3.0 the openstack on subcloud works well . central cloud does not support openstack . Thanks. BR Austin Sun. From: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Sent: Friday, April 17, 2020 11:01 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [starlingx] cinder bootstrap failed on subcloud I thought that dist-cloud sub-nodes are Kubernetes only – without OpenStack support… brucej From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Friday, April 17, 2020 5:45 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
Hi Andy : Thanks your reply. Some output currently , the result of 'openstack endpoint list' in cinder-bootstrap is [root@cinder-bootstrap-rtxf7 /]# openstack endpoint list | grep identity | 0da674cb1d384ae0999d02b0ababd5c9 | subcloud1 | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 | | 6eabcff1e83e45e3911940cd3195e09c | subcloud1 | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local:80/v3 | | e6f5720ee62c4c1ab378fe20c7d6716f | subcloud1 | keystone | identity | True | admin | http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3 | controller-0:~$ kubectl get ingress --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE openstack cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h openstack glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h openstack keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h openstack keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h openstack keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h openstack osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h ' kubectl get ingress -openstack' run with error But controller-0:~$ kubectl get ingress -n openstack NAME HOSTS ADDRESS PORTS AGE cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h But keystone-admin.openstack.svc.cluster.subcloud1 cannot be found from host or container : controller-0:~$ curl http://keystone-admin.openstack.svc.cluster.subcloud1 curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Name or service not known [root@cinder-bootstrap-rtxf7 /]# curl http://keystone-admin.openstack.svc.cluster.subcloud1 curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Unknown error Thanks. BR Austin Sun. From: Ning, Antai (Andy) <Andy.Ning@windriver.com> Sent: Wednesday, April 29, 2020 9:07 PM To: Sun, Austin <austin.sun@intel.com>; starlingx-discuss@lists.starlingx.io Cc: Jones, Bruce E <bruce.e.jones@intel.com> Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Austin, What's the output of "openstack endpoint list" on containerized openstack like? keystone-admin should be show up as identity's admin endpoint .... if not, that could be a bug. Also what's the output of "kubectl get ingress --all-namespaces" or "kubectl get ingress --openstack"? I want to see whether the ingress for keystone-admin is created or not. Andy ________________________________ From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Tuesday, April 28, 2020 4:34 AM To: Ning, Antai (Andy) <Andy.Ning@windriver.com<mailto:Andy.Ning@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Subject: FW: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Andy: I deployed dist-cloud in virtual env based on 20200414, but openstack application on subcloud deployed failed .
From cinder-api po logs:
: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1 controllers This url seems override from helm keystone: host_fqdn_override: admin: host: keystone-admin.openstack.svc.cluster.subcloud1 public: {} and introduced in commit [1] by you. is these any additional config needed to bring up openstack on subcloud besides the deploy guide [2]? [1] https://review.opendev.org/#/c/697361/1 [2] https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud... Thanks. BR Austin Sun. From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Saturday, April 18, 2020 9:06 AM To: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Bruce: No, for stx.3.0 the openstack on subcloud works well . central cloud does not support openstack . Thanks. BR Austin Sun. From: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Sent: Friday, April 17, 2020 11:01 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [starlingx] cinder bootstrap failed on subcloud I thought that dist-cloud sub-nodes are Kubernetes only - without OpenStack support... brucej From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Friday, April 17, 2020 5:45 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
It looks like a dns issue. "keystone-admin.openstack.svc.cluster.subcloud1" is not resolvable. coredns should be able to resolve this FQDN, but looks like it can't. So it can be bug. You may want to raise a LP for it. Thanks, Andy ________________________________ From: Sun, Austin <austin.sun@intel.com> Sent: Wednesday, April 29, 2020 10:15 AM To: Ning, Antai (Andy) <Andy.Ning@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io> Cc: Jones, Bruce E <bruce.e.jones@intel.com> Subject: RE: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Andy : Thanks your reply. Some output currently , the result of ‘openstack endpoint list’ in cinder-bootstrap is [root@cinder-bootstrap-rtxf7 /]# openstack endpoint list | grep identity | 0da674cb1d384ae0999d02b0ababd5c9 | subcloud1 | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 | | 6eabcff1e83e45e3911940cd3195e09c | subcloud1 | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local:80/v3 | | e6f5720ee62c4c1ab378fe20c7d6716f | subcloud1 | keystone | identity | True | admin | http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3 | controller-0:~$ kubectl get ingress --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE openstack cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h openstack glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h openstack keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h openstack keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h openstack keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h openstack osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h ‘ kubectl get ingress –openstack’ run with error But controller-0:~$ kubectl get ingress -n openstack NAME HOSTS ADDRESS PORTS AGE cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h But keystone-admin.openstack.svc.cluster.subcloud1 cannot be found from host or container : controller-0:~$ curl http://keystone-admin.openstack.svc.cluster.subcloud1 curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Name or service not known [root@cinder-bootstrap-rtxf7 /]# curl http://keystone-admin.openstack.svc.cluster.subcloud1 curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Unknown error Thanks. BR Austin Sun. From: Ning, Antai (Andy) <Andy.Ning@windriver.com> Sent: Wednesday, April 29, 2020 9:07 PM To: Sun, Austin <austin.sun@intel.com>; starlingx-discuss@lists.starlingx.io Cc: Jones, Bruce E <bruce.e.jones@intel.com> Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Austin, What's the output of "openstack endpoint list" on containerized openstack like? keystone-admin should be show up as identity's admin endpoint .... if not, that could be a bug. Also what's the output of "kubectl get ingress --all-namespaces" or "kubectl get ingress --openstack"? I want to see whether the ingress for keystone-admin is created or not. Andy ________________________________ From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Tuesday, April 28, 2020 4:34 AM To: Ning, Antai (Andy) <Andy.Ning@windriver.com<mailto:Andy.Ning@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Subject: FW: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Andy: I deployed dist-cloud in virtual env based on 20200414, but openstack application on subcloud deployed failed .
From cinder-api po logs:
: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1 controllers This url seems override from helm keystone: host_fqdn_override: admin: host: keystone-admin.openstack.svc.cluster.subcloud1 public: {} and introduced in commit [1] by you. is these any additional config needed to bring up openstack on subcloud besides the deploy guide [2]? [1] https://review.opendev.org/#/c/697361/1 [2] https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud... Thanks. BR Austin Sun. From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Saturday, April 18, 2020 9:06 AM To: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Bruce: No, for stx.3.0 the openstack on subcloud works well . central cloud does not support openstack . Thanks. BR Austin Sun. From: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Sent: Friday, April 17, 2020 11:01 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [starlingx] cinder bootstrap failed on subcloud I thought that dist-cloud sub-nodes are Kubernetes only – without OpenStack support… brucej From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Friday, April 17, 2020 5:45 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
Thanks. Here it is : https://bugs.launchpad.net/starlingx/+bug/1875914 Thanks. BR Austin Sun. From: Ning, Antai (Andy) <Andy.Ning@windriver.com> Sent: Wednesday, April 29, 2020 11:05 PM To: Sun, Austin <austin.sun@intel.com>; starlingx-discuss@lists.starlingx.io Cc: Jones, Bruce E <bruce.e.jones@intel.com> Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud It looks like a dns issue. "keystone-admin.openstack.svc.cluster.subcloud1" is not resolvable. coredns should be able to resolve this FQDN, but looks like it can't. So it can be bug. You may want to raise a LP for it. Thanks, Andy ________________________________ From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Wednesday, April 29, 2020 10:15 AM To: Ning, Antai (Andy) <Andy.Ning@windriver.com<mailto:Andy.Ning@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Subject: RE: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Andy : Thanks your reply. Some output currently , the result of 'openstack endpoint list' in cinder-bootstrap is [root@cinder-bootstrap-rtxf7 /]# openstack endpoint list | grep identity | 0da674cb1d384ae0999d02b0ababd5c9 | subcloud1 | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 | | 6eabcff1e83e45e3911940cd3195e09c | subcloud1 | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local:80/v3 | | e6f5720ee62c4c1ab378fe20c7d6716f | subcloud1 | keystone | identity | True | admin | http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3 | controller-0:~$ kubectl get ingress --all-namespaces NAMESPACE NAME HOSTS ADDRESS PORTS AGE openstack cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h openstack glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h openstack keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h openstack keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h openstack keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h openstack osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h ' kubectl get ingress -openstack' run with error But controller-0:~$ kubectl get ingress -n openstack NAME HOSTS ADDRESS PORTS AGE cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h But keystone-admin.openstack.svc.cluster.subcloud1 cannot be found from host or container : controller-0:~$ curl http://keystone-admin.openstack.svc.cluster.subcloud1 curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Name or service not known [root@cinder-bootstrap-rtxf7 /]# curl http://keystone-admin.openstack.svc.cluster.subcloud1 curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Unknown error Thanks. BR Austin Sun. From: Ning, Antai (Andy) <Andy.Ning@windriver.com<mailto:Andy.Ning@windriver.com>> Sent: Wednesday, April 29, 2020 9:07 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Cc: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Austin, What's the output of "openstack endpoint list" on containerized openstack like? keystone-admin should be show up as identity's admin endpoint .... if not, that could be a bug. Also what's the output of "kubectl get ingress --all-namespaces" or "kubectl get ingress --openstack"? I want to see whether the ingress for keystone-admin is created or not. Andy ________________________________ From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Tuesday, April 28, 2020 4:34 AM To: Ning, Antai (Andy) <Andy.Ning@windriver.com<mailto:Andy.Ning@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>> Cc: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Subject: FW: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Andy: I deployed dist-cloud in virtual env based on 20200414, but openstack application on subcloud deployed failed .
From cinder-api po logs:
: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1 controllers This url seems override from helm keystone: host_fqdn_override: admin: host: keystone-admin.openstack.svc.cluster.subcloud1 public: {} and introduced in commit [1] by you. is these any additional config needed to bring up openstack on subcloud besides the deploy guide [2]? [1] https://review.opendev.org/#/c/697361/1 [2] https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud... Thanks. BR Austin Sun. From: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>> Sent: Saturday, April 18, 2020 9:06 AM To: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi Bruce: No, for stx.3.0 the openstack on subcloud works well . central cloud does not support openstack . Thanks. BR Austin Sun. From: Jones, Bruce E <bruce.e.jones@intel.com<mailto:bruce.e.jones@intel.com>> Sent: Friday, April 17, 2020 11:01 PM To: Sun, Austin <austin.sun@intel.com<mailto:austin.sun@intel.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: RE: [starlingx] cinder bootstrap failed on subcloud I thought that dist-cloud sub-nodes are Kubernetes only - without OpenStack support... brucej From: Sun, Austin [mailto:austin.sun@intel.com] Sent: Friday, April 17, 2020 5:45 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud Hi community: I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed. The logs are [root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store The server is currently unavailable. Please try again at a later time.<br /><br /> The Keystone service is temporarily unavailable. (HTTP 503) But openstack endpoint list works fine on this pod Does any one meet same issue on recently ? Thanks. BR Austin Sun.
participants (3)
-
Jones, Bruce E
-
Ning, Antai (Andy)
-
Sun, Austin