[Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud
Ning, Antai (Andy)
Andy.Ning at windriver.com
Wed Apr 29 15:04:39 UTC 2020
It looks like a dns issue. "keystone-admin.openstack.svc.cluster.subcloud1" is not resolvable.
coredns should be able to resolve this FQDN, but looks like it can't. So it can be bug. You may want to raise a LP for it.
Thanks,
Andy
________________________________
From: Sun, Austin <austin.sun at intel.com>
Sent: Wednesday, April 29, 2020 10:15 AM
To: Ning, Antai (Andy) <Andy.Ning at windriver.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Cc: Jones, Bruce E <bruce.e.jones at intel.com>
Subject: RE: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud
Hi Andy :
Thanks your reply.
Some output
currently , the result of ‘openstack endpoint list’ in cinder-bootstrap is
[root at cinder-bootstrap-rtxf7 /]# openstack endpoint list | grep identity
| 0da674cb1d384ae0999d02b0ababd5c9 | subcloud1 | keystone | identity | True | internal | http://keystone-api.openstack.svc.cluster.local:5000/v3 |
| 6eabcff1e83e45e3911940cd3195e09c | subcloud1 | keystone | identity | True | public | http://keystone.openstack.svc.cluster.local:80/v3 |
| e6f5720ee62c4c1ab378fe20c7d6716f | subcloud1 | keystone | identity | True | admin | http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3 |
controller-0:~$ kubectl get ingress --all-namespaces
NAMESPACE NAME HOSTS ADDRESS PORTS AGE
openstack cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h
openstack glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h
openstack keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h
openstack keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h
openstack keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h
openstack keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h
openstack openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h
openstack osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h
‘ kubectl get ingress –openstack’ run with error
But
controller-0:~$ kubectl get ingress -n openstack
NAME HOSTS ADDRESS PORTS AGE
cinder cinder,cinder.openstack,cinder.openstack.svc.cluster.local 80 31h
glance glance,glance.openstack,glance.openstack.svc.cluster.local 80 31h
keystone keystone,keystone.openstack,keystone.openstack.svc.cluster.local 80 31h
keystone-admin keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local 80 31h
keystone-admin-cluster-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h
keystone-admin-namespace-fqdn keystone-admin.openstack.svc.cluster.subcloud1 80 31h
openstack-osh-openstack-ingress *.openstack.svc.cluster.local 80 31h
osh-openstac-mgr-e50215 osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local 80 31h
But keystone-admin.openstack.svc.cluster.subcloud1 cannot be found from host or container :
controller-0:~$ curl http://keystone-admin.openstack.svc.cluster.subcloud1
curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Name or service not known
[root at cinder-bootstrap-rtxf7 /]# curl http://keystone-admin.openstack.svc.cluster.subcloud1
curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Unknown error
Thanks.
BR
Austin Sun.
From: Ning, Antai (Andy) <Andy.Ning at windriver.com>
Sent: Wednesday, April 29, 2020 9:07 PM
To: Sun, Austin <austin.sun at intel.com>; starlingx-discuss at lists.starlingx.io
Cc: Jones, Bruce E <bruce.e.jones at intel.com>
Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud
Hi Austin,
What's the output of "openstack endpoint list" on containerized openstack like?
keystone-admin should be show up as identity's admin endpoint .... if not, that could be a bug.
Also what's the output of "kubectl get ingress --all-namespaces" or "kubectl get ingress --openstack"? I want to see whether the ingress for keystone-admin is created or not.
Andy
________________________________
From: Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Sent: Tuesday, April 28, 2020 4:34 AM
To: Ning, Antai (Andy) <Andy.Ning at windriver.com<mailto:Andy.Ning at windriver.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>
Subject: FW: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud
Hi Andy:
I deployed dist-cloud in virtual env based on 20200414, but openstack application on subcloud deployed failed .
>From cinder-api po logs:
>: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))
This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1 controllers
This url seems override from helm keystone:
host_fqdn_override:
admin:
host: keystone-admin.openstack.svc.cluster.subcloud1
public: {}
and introduced in commit [1] by you.
is these any additional config needed to bring up openstack on subcloud besides the deploy guide [2]?
[1] https://review.opendev.org/#/c/697361/1
[2] https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud/index.html
Thanks.
BR
Austin Sun.
From: Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Sent: Saturday, April 18, 2020 9:06 AM
To: Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud
Hi Bruce:
No, for stx.3.0 the openstack on subcloud works well . central cloud does not support openstack .
Thanks.
BR
Austin Sun.
From: Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>
Sent: Friday, April 17, 2020 11:01 PM
To: Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [starlingx] cinder bootstrap failed on subcloud
I thought that dist-cloud sub-nodes are Kubernetes only – without OpenStack support…
brucej
From: Sun, Austin [mailto:austin.sun at intel.com]
Sent: Friday, April 17, 2020 5:45 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud
Hi community:
I deployed dist-cloud in virtual env (20200414) , but the openstack application applied failed. The reason is cinder-bootstrap job failed.
The logs are
[root at cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store
The server is currently unavailable. Please try again at a later time.<br /><br />
The Keystone service is temporarily unavailable.
(HTTP 503)
But openstack endpoint list works fine on this pod
Does any one meet same issue on recently ?
Thanks.
BR
Austin Sun.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200429/a4bd7f0b/attachment-0001.html>
More information about the Starlingx-discuss
mailing list