Hi All

 

“……… after running Application-apply, the system takes about 8 to 10 minutes to stabilize”

 

This is something expected on StarlingX deployment? If this is the case  should be great to inform the user that system is not ready, maybe setting the node “Degraded” or something like this.

 

Regards,

José

 

From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com]
Sent: Thursday, April 11, 2019 6:28 PM
To: Khalil, Ghada <Ghada.Khalil@windriver.com>; Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; starlingx-discuss@lists.starlingx.io
Cc: Liu, Yang <yang.liu@windriver.com>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

 

Hello Ghada,

 

we double checked the error and it does not appear anymore. We’ve found that after running Application-apply, the system takes about 8 to 10 minutes to stabilize after that, instances can be created correctly.

 

Regards

Maria G.

 

From: Khalil, Ghada [mailto:Ghada.Khalil@windriver.com]
Sent: Thursday, April 11, 2019 1:10 PM
To: Lemus Contreras, Cristopher J <cristopher.j.lemus.contreras@intel.com>; Perez Ibarra, Maria G <maria.g.perez.ibarra@intel.com>; starlingx-discuss@lists.starlingx.io
Cc: Liu, Yang <yang.liu@windriver.com>
Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

 

This looks like a different issue unrelated to the original issue reported in

https://bugs.launchpad.net/starlingx/+bug/1823275

https://bugs.launchpad.net/starlingx/+bug/1821938

 

 

From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com]
Sent: Wednesday, April 10, 2019 11:12 PM
To: Khalil, Ghada; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io
Cc: Liu, Yang
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

 

Hi Ghada,

 

Logs on nova pod are not showing any ERROR, just this warning (log shows that is fixed on the next line):

 

2019-04-10 12:52:47,173.173 229781 WARNING nova.conductor.api [req-c557fabd-b379-4dda-b64e-981e71b530bf - - - - -] Timed out waiting for nova-conductor.  Is it running? Or did this service start before nova-conductor?  Reattempting establishment of nova-conductor connection...: MessagingTimeout: Timed out waiting for a reply to message ID af6fd23a7b6a476985949f42947f0d4e

2019-04-10 12:52:47,192.192 229781 INFO nova.conductor.api [req-c557fabd-b379-4dda-b64e-981e71b530bf - - - - -] nova-conductor connection established successfully

 

I’m not sure about the output of  nova hypervisor-list

 

DEBUG (shell:812) internalURL endpoint for compute service in RegionOne region not found

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 810, in main

    OpenStackComputeShell().main(argv)

  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 678, in main

    api_version = api_versions.discover_version(self.cs, api_version)

  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 261, in discover_version

    client)

  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 242, in _get_server_version_range

    version = client.versions.get_current()

  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 70, in get_current

    return self._get_current()

  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 52, in _get_current

    url = "%s" % self.api.client.get_endpoint()

  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 271, in get_endpoint

    return self.session.get_endpoint(auth or self.auth, **kwargs)

  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1139, in get_endpoint

    return auth.get_endpoint(self, **kwargs)

  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint

    allow_version_hack=allow_version_hack, **kwargs)

  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 279, in get_endpoint_data

    service_name=service_name)

  File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for

    raise exceptions.EndpointNotFound(msg)

EndpointNotFound: internalURL endpoint for compute service in RegionOne region not found

 

Regards,

 

Cristopher Lemus

 

 

From: "Khalil, Ghada" <Ghada.Khalil@windriver.com>
Date: Wednesday, April 10, 2019 at 9:50 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras@intel.com>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>
Cc: "Liu, Yang" <yang.liu@windriver.com>
Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

 

Hi Christopher,

These images look correct to me, but I’ll ask Yang to confirm the images she tested with tomorrow.

 

Can you confirm that nova is still returning the same error as was originally reported in https://bugs.launchpad.net/starlingx/+bug/1823275

 

nova is returning this error:
2019-04-04 16:22:30,902.902 168396 ERROR nova.compute.manager [req-b3bffaba-a62a-4e56-be63-3362c51a36df - - - - -] Error updating resources for node compute-0.: PciDeviceNotFoundById: PCI device 0000:b3:02.3 not found

 

Can you also list the output of the nova hypervisor cmd on the compute node?

nova hypervisor-list

 

Thanks,

Ghada

 

From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras@intel.com]
Sent: Wednesday, April 10, 2019 10:40 PM
To: Khalil, Ghada; Perez Ibarra, Maria G; starlingx-discuss@lists.starlingx.io
Cc: Liu, Yang
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

 

Hi Ghada,

 

These are the starlingx nova images used on the servers where the sanity was executed:

 

controller-0:~# docker images |grep starlingx/stx-nova

192.168.100.60/starlingx/stx-nova                                                   master-centos-stable-latest                6c8a3a356110        30 hours ago        1.18GB

192.168.204.2:9001/docker.io/starlingx/stx-nova                                     master-centos-stable-latest                6c8a3a356110        30 hours ago        1.18GB

192.168.100.60/starlingx/stx-nova-api-proxy                                         master-centos-stable-latest                c5853883d561        31 hours ago        589MB

192.168.204.2:9001/docker.io/starlingx/stx-nova-api-proxy                           master-centos-stable-latest                c5853883d561        31 hours ago        589MB

controller-0:~#

 

 

For stx-nova, id: 6c8a3a356110           and for stx-nova-api-proxy, id: c5853883d561       

 

 

I manually verified the images on our local registry, and tried to download the latest version:

 

[root@registry ~]# docker pull starlingx/stx-nova:master-centos-stable-latest

master-centos-stable-latest: Pulling from starlingx/stx-nova

Digest: sha256:e9fecb6998ad0cf0a6621a8c5c2422378d0972c73ed80b636dcbd4bb5183794e

Status: Image is up to date for starlingx/stx-nova:master-centos-stable-latest

[root@registry ~]# docker pull starlingx/stx-nova-api-proxy:master-centos-stable-latest

master-centos-stable-latest: Pulling from starlingx/stx-nova-api-proxy

Digest: sha256:06c9bf3546eeec5025e62ccf00c24dbaa20e6d79614d89709c0a71ffa639c5f8

Status: Image is up to date for starlingx/stx-nova-api-proxy:master-centos-stable-latest

[root@registry ~]# docker images |grep starlingx/stx-nova

registry.zpn.intel.com/starlingx/stx-nova                                               master-centos-stable-latest                6c8a3a356110        28 hours ago        1.18GB

starlingx/stx-nova                                                                      master-centos-stable-latest                6c8a3a356110        28 hours ago        1.18GB

starlingx/stx-nova-api-proxy                                                            master-centos-stable-latest                c5853883d561        29 hours ago        589MB

registry.zpn.intel.com/starlingx/stx-nova-api-proxy                                     master-centos-stable-latest                c5853883d561        29 hours ago        589MB

 

Images are up to date, and the timestamp matches what is logged in dockerhub: https://hub.docker.com/r/starlingx/stx-nova/tags . I assume that those are the images according to the comments on https://bugs.launchpad.net/starlingx/+bug/1821938, are there other nova images that we need to pull? I did a quick check and it looks like all of our local images (the ones deployed to the sanity servers) are updated.

 

Thanks & Regards,

 

Cristopher Lemus

 

 

 

 

From: "Khalil, Ghada" <Ghada.Khalil@windriver.com>
Date: Wednesday, April 10, 2019 at 8:27 PM
To: "Perez Ibarra, Maria G" <maria.g.perez.ibarra@intel.com>, "starlingx-discuss@lists.starlingx.io" <starlingx-discuss@lists.starlingx.io>
Cc: "Liu, Yang" <yang.liu@windriver.com>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

 

Hi Maria,

Regarding https://bugs.launchpad.net/starlingx/+bug/1821938, Yang successfully verified that the issue is resolved using the same load you list below (see her note in the LP).

Can you confirm that you are using the stable docker images built yesterday? The fix should be included in the nova image.

 

Thanks,

Ghada

 

From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra@intel.com]
Sent: Wednesday, April 10, 2019 8:59 PM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

 

Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-10 (link)

 

Status: YELLOW

 

===========================================

 

Sanity Test is executed in a Containers – Bare Metal Environment

 

AIO - Simplex

 

Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [FAIL] | 10 TCs FAIL

Sanity Platform   07 TCs [FAIL] | 03 TCs FAIL

 

TOTAL: 57 TCS [PASS : 44] [Fail : 13]

 

AIO – Duplex

 

Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] | 10 TCs FAIL

Sanity Platform   05 TCs [PASS] | 01 TCs FAIL

 

TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 TCs]

 

Standard - Local Storage (2+2)

 

Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] | 10 TCs FAIL

Sanity Platform   05 TCs [PASS] | 01 TCs FAIL

 

TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 TCs]

 

Standard - Dedicated Storage (2+2+2)

 

Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] | 10 TCs FAIL

Sanity Platform   05 TCs [PASS] | 01 TCs FAIL

 

TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 Tcs]

 

 

 

Sanity Test is executed in a Containers - Virtual Environment

 

AIO - Simplex

 

Setup             04 TCs [PASS] 

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]

 

TOTAL: [ 61 TCs PASS ]

 

 

AIO - Duplex

 

Setup             04 TCs [PASS] 

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [PASS]

Sanity Platform   07 TCs [PASS]

 

TOTAL: [ 61 TCs PASS ]

 

 

Standard – Local Storage

 

Setup             04 TCs [PASS] 

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS]

Sanity Platform   05 TCs [PASS]

 

TOTAL: [ 61 TCs PASS ]

 

 

 

--------------------------------------------------------------------------------

No nova hypervisor can be enabled on workers with QAT devices  https://bugs.launchpad.net/starlingx/+bug/1821938                                                                                                                                                                                 

 

For more detail of the tests: https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack

 

 

Regards

Maria G.