<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
It looks like a dns issue. "keystone-admin.openstack.svc.cluster.subcloud1" is not resolvable.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
coredns should be able to resolve this FQDN, but looks like it can't. So it can be bug. You may want to raise a LP for it.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Andy<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<br>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Sun, Austin <austin.sun@intel.com><br>
<b>Sent:</b> Wednesday, April 29, 2020 10:15 AM<br>
<b>To:</b> Ning, Antai (Andy) <Andy.Ning@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io><br>
<b>Cc:</b> Jones, Bruce E <bruce.e.jones@intel.com><br>
<b>Subject:</b> RE: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud</font>
<div> </div>
</div>
<style>
<!--
@font-face
        {font-family:SimSun}
@font-face
        {font-family:"Cambria Math"}
@font-face
        {font-family:Calibri}
@font-face
        {font-family:SimSun}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
        {color:#0563C1;
        text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
        {color:#954F72;
        text-decoration:underline}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xmsonormal0, li.x_xmsonormal0, div.x_xmsonormal0
        {margin-right:0in;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xmsochpdefault, li.x_xmsochpdefault, div.x_xmsochpdefault
        {margin-right:0in;
        margin-left:0in;
        font-size:10.0pt;
        font-family:"Calibri",sans-serif}
span.x_xmsohyperlink
        {color:#0563C1;
        text-decoration:underline}
span.x_xmsohyperlinkfollowed
        {color:#954F72;
        text-decoration:underline}
span.x_xemailstyle18
        {font-family:"Calibri",sans-serif;
        color:windowtext}
span.x_xemailstyle19
        {font-family:"Calibri",sans-serif;
        color:#1F497D}
span.x_xemailstyle20
        {font-family:"Calibri",sans-serif;
        color:windowtext}
span.x_xemailstyle22
        {font-family:"Calibri",sans-serif;
        color:windowtext}
span.x_EmailStyle29
        {font-family:"Calibri",sans-serif;
        color:windowtext}
.x_MsoChpDefault
        {font-size:10.0pt}
@page WordSection1
        {margin:1.0in 1.25in 1.0in 1.25in}
div.x_WordSection1
        {}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="x_WordSection1">
<p class="x_MsoNormal">Hi Andy :</p>
<p class="x_MsoNormal">   Thanks your reply. </p>
<p class="x_MsoNormal">Some output </p>
<p class="x_MsoNormal">currently , the result of ‘openstack endpoint list’ in cinder-bootstrap is
</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">[root@cinder-bootstrap-rtxf7 /]# openstack endpoint list | grep identity</p>
<p class="x_MsoNormal">| 0da674cb1d384ae0999d02b0ababd5c9 | subcloud1 | keystone     | identity     | True    | internal  | http://keystone-api.openstack.svc.cluster.local:5000/v3                   |</p>
<p class="x_MsoNormal">| 6eabcff1e83e45e3911940cd3195e09c | subcloud1 | keystone     | identity     | True    | public    | http://keystone.openstack.svc.cluster.local:80/v3                         |</p>
<p class="x_MsoNormal">| e6f5720ee62c4c1ab378fe20c7d6716f | subcloud1 | keystone     | identity     | True    | admin     | http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3               |</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">controller-0:~$ kubectl get ingress --all-namespaces</p>
<p class="x_MsoNormal">NAMESPACE   NAME                              HOSTS                                                                                                           ADDRESS   PORTS   AGE</p>
<p class="x_MsoNormal">openstack   cinder                            cinder,cinder.openstack,cinder.openstack.svc.cluster.local                                                                80      31h</p>
<p class="x_MsoNormal">openstack   glance                            glance,glance.openstack,glance.openstack.svc.cluster.local                                                                80      31h</p>
<p class="x_MsoNormal">openstack   keystone                          keystone,keystone.openstack,keystone.openstack.svc.cluster.local                                                          80      31h</p>
<p class="x_MsoNormal">openstack   keystone-admin                    keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local                                        80      31h</p>
<p class="x_MsoNormal">openstack   keystone-admin-cluster-fqdn       keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h</p>
<p class="x_MsoNormal">openstack   keystone-admin-namespace-fqdn     keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h</p>
<p class="x_MsoNormal">openstack   openstack-osh-openstack-ingress   *.openstack.svc.cluster.local                                                                                             80      31h</p>
<p class="x_MsoNormal">openstack   osh-openstac-mgr-e50215           osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local             80      31h</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">‘ kubectl get ingress –openstack’ run with error
</span></p>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">But </span></p>
<p class="x_MsoNormal">controller-0:~$ kubectl get ingress -n openstack</p>
<p class="x_MsoNormal">NAME                              HOSTS                                                                                                           ADDRESS   PORTS   AGE</p>
<p class="x_MsoNormal">cinder                            cinder,cinder.openstack,cinder.openstack.svc.cluster.local                                                                80      31h</p>
<p class="x_MsoNormal">glance                            glance,glance.openstack,glance.openstack.svc.cluster.local                                                                80      31h</p>
<p class="x_MsoNormal">keystone                          keystone,keystone.openstack,keystone.openstack.svc.cluster.local                                                          80      31h</p>
<p class="x_MsoNormal">keystone-admin                    keystone-admin,keystone-admin.openstack,keystone-admin.openstack.svc.cluster.local                                        80      31h</p>
<p class="x_MsoNormal">keystone-admin-cluster-fqdn       keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h</p>
<p class="x_MsoNormal">keystone-admin-namespace-fqdn     keystone-admin.openstack.svc.cluster.subcloud1                                                                            80      31h</p>
<p class="x_MsoNormal">openstack-osh-openstack-ingress   *.openstack.svc.cluster.local                                                                                             80      31h</p>
<p class="x_MsoNormal">osh-openstac-mgr-e50215           osh-openstac-mgr-e50215,osh-openstac-mgr-e50215.openstack,osh-openstac-mgr-e50215.openstack.svc.cluster.local             80      31h</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">But keystone-admin.openstack.svc.cluster.subcloud1 cannot be found from host or container :</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">controller-0:~$ curl http://keystone-admin.openstack.svc.cluster.subcloud1</p>
<p class="x_MsoNormal">curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Name or service not known</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">[root@cinder-bootstrap-rtxf7 /]# curl http://keystone-admin.openstack.svc.cluster.subcloud1</p>
<p class="x_MsoNormal">curl: (6) Could not resolve host: keystone-admin.openstack.svc.cluster.subcloud1; Unknown error</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Thanks.</p>
<p class="x_MsoNormal">BR<br>
Austin Sun. </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b>From:</b> Ning, Antai (Andy) <Andy.Ning@windriver.com> <br>
<b>Sent:</b> Wednesday, April 29, 2020 9:07 PM<br>
<b>To:</b> Sun, Austin <austin.sun@intel.com>; starlingx-discuss@lists.starlingx.io<br>
<b>Cc:</b> Jones, Bruce E <bruce.e.jones@intel.com><br>
<b>Subject:</b> Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Hi Austin,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">What's the output of "openstack endpoint list" on containerized openstack like?
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">keystone-admin should be show up as identity's admin endpoint .... if not, that could be a bug.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Also what's the output of "kubectl get ingress --all-namespaces" or "kubectl get ingress --openstack"? I want to see whether the ingress for keystone-admin is created or not.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Andy  </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Sun, Austin <<a href="mailto:austin.sun@intel.com">austin.sun@intel.com</a>><br>
<b>Sent:</b> Tuesday, April 28, 2020 4:34 AM<br>
<b>To:</b> Ning, Antai (Andy) <<a href="mailto:Andy.Ning@windriver.com">Andy.Ning@windriver.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a> <<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Cc:</b> Jones, Bruce E <<a href="mailto:bruce.e.jones@intel.com">bruce.e.jones@intel.com</a>><br>
<b>Subject:</b> FW: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud</span>
</p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal">Hi Andy:</p>
<p class="x_xmsonormal">    I deployed dist-cloud in virtual env  based on 20200414, but openstack application on subcloud deployed failed . 
</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">From cinder-api po logs:</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">>: Failed to establish a new connection: [Errno -2] Name or service not known',)): ConnectFailure: Unable to establish connection to
<a href="http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens">
http://keystone-admin.openstack.svc.cluster.subcloud1:80/v3/auth/tokens</a>: HTTPConnectionPool(host='keystone-admin.openstack.svc.cluster.subcloud1', port=80): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection
 object at 0x7fc6beac6d10>: Failed to establish a new connection: [Errno -2] Name or service not known',))</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">This keystone-admin.openstack.svc.cluster.subcloud1 can not be found in subcloud1  controllers
</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">This url seems override from  helm keystone:</p>
<p class="x_xmsonormal">host_fqdn_override:                                      </p>
<p class="x_xmsonormal">  admin:                                                 </p>
<p class="x_xmsonormal">    host: keystone-admin.openstack.svc.cluster.subcloud1 </p>
<p class="x_xmsonormal">  public: {}                                             </p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">and introduced in commit [1]  by you. </p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">is these any additional config needed to bring up openstack on subcloud  besides the deploy guide [2]?
</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">[1] <a href="https://review.opendev.org/#/c/697361/1">https://review.opendev.org/#/c/697361/1</a></p>
<p class="x_xmsonormal">[2] <a href="https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud/index.html">
https://docs.starlingx.io/deploy_install_guides/r3_release/distributed_cloud/index.html</a>
</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Thanks.</p>
<p class="x_xmsonormal">BR<br>
Austin Sun. </p>
<p class="x_xmsonormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_xmsonormal"><b>From:</b> Sun, Austin <<a href="mailto:austin.sun@intel.com">austin.sun@intel.com</a>>
<br>
<b>Sent:</b> Saturday, April 18, 2020 9:06 AM<br>
<b>To:</b> Jones, Bruce E <<a href="mailto:bruce.e.jones@intel.com">bruce.e.jones@intel.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> Re: [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud</p>
</div>
</div>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Hi Bruce:</p>
<p class="x_xmsonormal">   No, for stx.3.0 the openstack on subcloud works well .  central cloud does not support openstack .
</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Thanks.</p>
<p class="x_xmsonormal">BR<br>
Austin Sun. </p>
<p class="x_xmsonormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_xmsonormal"><b>From:</b> Jones, Bruce E <<a href="mailto:bruce.e.jones@intel.com">bruce.e.jones@intel.com</a>>
<br>
<b>Sent:</b> Friday, April 17, 2020 11:01 PM<br>
<b>To:</b> Sun, Austin <<a href="mailto:austin.sun@intel.com">austin.sun@intel.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> RE: [starlingx] cinder bootstrap failed on subcloud</p>
</div>
</div>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal"><span style="color:#1F497D">I thought that dist-cloud sub-nodes are Kubernetes only – without OpenStack support…</span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<p class="x_xmsonormal"><span style="color:#1F497D">        brucej</span></p>
<p class="x_xmsonormal"><span style="color:#1F497D"> </span></p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_xmsonormal"><b>From:</b> Sun, Austin [<a href="mailto:austin.sun@intel.com">mailto:austin.sun@intel.com</a>]
<br>
<b>Sent:</b> Friday, April 17, 2020 5:45 AM<br>
<b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> [Starlingx-discuss] [starlingx] cinder bootstrap failed on subcloud</p>
</div>
</div>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Hi community:</p>
<p class="x_xmsonormal" style="text-indent:10.2pt">I deployed dist-cloud in virtual env (20200414) ,   but the openstack application applied failed. The reason is cinder-bootstrap job failed.
</p>
<p class="x_xmsonormal" style="text-indent:10.2pt"> </p>
<p class="x_xmsonormal" style="text-indent:10.2pt">The logs are </p>
<p class="x_xmsonormal" style="text-indent:10.2pt">[root@cinder-bootstrap-rkp7j /]# openstack volume type show ceph-store</p>
<p class="x_xmsonormal" style="text-indent:10.2pt">The server is currently unavailable. Please try again at a later time.<br /><br /></p>
<p class="x_xmsonormal" style="text-indent:10.2pt">The Keystone service is temporarily unavailable.</p>
<p class="x_xmsonormal" style="text-indent:10.2pt"> </p>
<p class="x_xmsonormal" style="text-indent:10.2pt">(HTTP 503)</p>
<p class="x_xmsonormal" style="text-indent:10.2pt"> </p>
<p class="x_xmsonormal" style="text-indent:10.2pt">But openstack endpoint list works fine on this pod
</p>
<p class="x_xmsonormal" style="text-indent:10.2pt">Does any one meet same issue on recently ? 
</p>
<p class="x_xmsonormal" style="text-indent:10.2pt"> </p>
<p class="x_xmsonormal" style="text-indent:10.2pt">Thanks.</p>
<p class="x_xmsonormal" style="text-indent:10.2pt">BR<br>
Austin Sun. </p>
</div>
</div>
</div>
</div>
</body>
</html>