[Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud

Ning, Antai (Andy) Andy.Ning at windriver.com
Wed Apr 29 15:04:39 UTC 2020


It looks like a dns issue. "keystone-admin.openstack.svc.cluster.subcloud1" is not resolvable.

coredns should be able to resolve this FQDN, but looks like it can't. So it can be bug. You may want to raise a LP for it.

Thanks,
Andy



________________________________
From: Sun, Austin <austin.sun at intel.com>
Sent: Wednesday, April 29, 2020 10:15 AM
To: Ning, Antai (Andy) <Andy.Ning at windriver.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Cc: Jones, Bruce E <bruce.e.jones at intel.com>
Subject: RE: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud


Hi Andy :

   Thanks your reply.

Some output

currently , the result of ‘openstack endpoint list’ in cinder-bootstrap is



[root at cinder-bootstrap-rtxf7 /]# openstack endpoint list | grep identity

| 0da674cb1d384ae0999d02b0ababd5c9 | subcloud1 | keystone     | identity     | True    | internal  | http://keystone-api.openstack.svc.cluster.local:5000/v3                   |

| 6eabcff1e83e45e3911940cd3195e09c | subcloud1 | keystone     | identity     | True    | public    | http://keystone.openstack.svc.cluster.local:80/v3                         |

| e6f5720ee62c4c1ab378fe20c7d6716f | subcloud1 | keystone     | identity     | True    | admin     | http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3               |



controller-0:~$ kubectl get ingress --all-namespaces

NAMESPACE   NAME                              HOSTS                                                                                                           ADDRESS   PORTS   AGE

openstack   cinder                            cinder,cinder.openstack,cinder.openstack.svc.cluster.local                                                                80      31h

openstack   glance                            glance,glance.openstack,glance.openstack.svc.cluster.local                                                                80      31h

openstack   keystone                          keystone,keystone.openstack,keystone.openstack.svc.cluster.local                                                          80      31h

openstack   keystone-admin                    keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local                                        80      31h

openstack   keystone-admin-cluster-fqdn       keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h

openstack   keystone-admin-namespace-fqdn     keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h

openstack   openstack-osh-openstack-ingress   *.openstack.svc.cluster.local                                                                                             80      31h

openstack   osh-openstac-mgr-e50215           osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local             80      31h





‘ kubectl get ingress –openstack’ run with error



But

controller-0:~$ kubectl get ingress -n openstack

NAME                              HOSTS                                                                                                           ADDRESS   PORTS   AGE

cinder                            cinder,cinder.openstack,cinder.openstack.svc.cluster.local                                                                80      31h

glance                            glance,glance.openstack,glance.openstack.svc.cluster.local                                                                80      31h

keystone                          keystone,keystone.openstack,keystone.openstack.svc.cluster.local                                                          80      31h

keystone-admin                    keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local                                        80      31h

keystone-admin-cluster-fqdn       keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h

keystone-admin-namespace-fqdn     keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h

openstack-osh-openstack-ingress   *.openstack.svc.cluster.local                                                                                             80      31h

osh-openstac-mgr-e50215           osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local             80      31h





But keystone-admin.openstack.svc.cluster.subcloud1 cannot be found from host or container :



controller-0:~$ curl http://keystone-admin.openstack.svc.cluster.subcloud1

curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Name or service not known



[root at cinder-bootstrap-rtxf7 /]# curl http://keystone-admin.openstack.svc.cluster.subcloud1

curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Unknown error



Thanks.

BR
Austin Sun.





From: Ning, Antai (Andy) <Andy.Ning at windriver.com>
Sent: Wednesday, April 29, 2020 9:07 PM
To: Sun, Austin <austin.sun at intel.com>; starlingx-discuss at lists.starlingx.io
Cc: Jones, Bruce E <bruce.e.jones at intel.com>
Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud



Hi Austin,



What's the output of "openstack endpoint list" on containerized openstack like?



keystone-admin should be show up as identity's admin endpoint .... if not, that could be a bug.



Also what's the output of "kubectl get ingress --all-namespaces" or "kubectl get ingress --openstack"? I want to see whether the ingress for keystone-admin is created or not.



Andy

________________________________

From: Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Sent: Tuesday, April 28, 2020 4:34 AM
To: Ning, Antai (Andy) <Andy.Ning at windriver.com<mailto:Andy.Ning at windriver.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io> <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>
Subject: FW: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud



Hi Andy:

    I deployed dist-cloud in virtual env  based on 20200414, but openstack application on subcloud deployed failed .



>From cinder-api po logs:



>: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))



This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1  controllers



This url seems override from  helm keystone:

host_fqdn_override:

  admin:

    host: keystone-admin.openstack.svc.cluster.subcloud1

  public: {}



and introduced in commit [1]  by you.



is these any additional config needed to bring up openstack on subcloud  besides the deploy guide [2]?



[1] https://review.opendev.org/#/c/697361/1

[2] https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud/index.html



Thanks.

BR
Austin Sun.



From: Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>
Sent: Saturday, April 18, 2020 9:06 AM
To: Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud



Hi Bruce:

   No, for stx.3.0 the openstack on subcloud works well .  central cloud does not support openstack .



Thanks.

BR
Austin Sun.



From: Jones, Bruce E <bruce.e.jones at intel.com<mailto:bruce.e.jones at intel.com>>
Sent: Friday, April 17, 2020 11:01 PM
To: Sun, Austin <austin.sun at intel.com<mailto:austin.sun at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: [starlingx] cinder bootstrap failed on subcloud



I thought that dist-cloud sub-nodes are Kubernetes only – without OpenStack support…



        brucej



From: Sun, Austin [mailto:austin.sun at intel.com]
Sent: Friday, April 17, 2020 5:45 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud



Hi community:

I deployed dist-cloud in virtual env (20200414) ,   but the openstack application applied failed. The reason is cinder-bootstrap job failed.



The logs are

[root at cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store

The server is currently unavailable. Please try again at a later time.<br /><br />

The Keystone service is temporarily unavailable.



(HTTP 503)



But openstack endpoint list works fine on this pod

Does any one meet same issue on recently ?



Thanks.

BR
Austin Sun.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20200429/a4bd7f0b/attachment-0001.html>


More information about the Starlingx-discuss mailing list