[Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

Liu, Yang yang.liu at windriver.com
Thu Apr 11 18:39:48 UTC 2019


Are you able to run any nova cmd Christopher?

$ source /etc/platform/openrc ; system host-device-list compute-0; export OS_AUTH_URL=http://keystone.openstack.svc.cluster.local/v3; nova hypervisor-list
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
| name             | address      | class id | vendor id | device id | class name                | vendor name                     | device name                            | numa_node | enabled |
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
| pci_0000_08_00_0 | 0000:08:00.0 | 0b4000   | 8086      | 0435      | Co-processor              | Intel Corporation               | DH895XCC Series QAT                    | 0         | True    |
| pci_0000_0c_00_0 | 0000:0c:00.0 | 030000   | 102b      | 0522      | VGA compatible controller | Matrox Electronics Systems Ltd. | MGA G200e [Pilot] ServerEngines (SEP1) | 0         | True    |
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
+--------------------------------------+---------------------+-------+---------+
| ID                                   | Hypervisor hostname | State | Status  |
+--------------------------------------+---------------------+-------+---------+
| 6136b80d-e2a9-4d34-97ce-d3818b305a7f | compute-0           | up    | enabled |
| d87821b4-4575-4238-b97c-b07589465127 | compute-2           | up    | enabled |
| f620e8ae-7353-4fbb-ad0d-f427b6c8bd00 | compute-3           | up    | enabled |
| a966bf58-6ce8-42d4-be8b-ea4453305644 | compute-1           | up    | enabled |
+--------------------------------------+---------------------+-------+---------+

Images are the same btw.
192.168.204.2:9001/docker.io/starlingx/stx-nova                                     master-centos-stable-latest                6c8a3a356110        2 days ago          1.18GB
starlingx/stx-nova                                                                  master-centos-stable-latest                6c8a3a356110        2 days ago          1.18GB
192.168.204.2:9001/docker.io/starlingx/stx-nova-api-proxy                           master-centos-stable-latest                c5853883d561        2 days ago          589MB
starlingx/stx-nova-api-proxy                                                        master-centos-stable-latest                c5853883d561        2 days ago          589MB

BR,
Yang

From: Khalil, Ghada
Sent: April-11-19 2:10 PM
To: Lemus Contreras, Cristopher J; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io
Cc: Liu, Yang
Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

This looks like a different issue unrelated to the original issue reported in
https://bugs.launchpad.net/starlingx/+bug/1823275
https://bugs.launchpad.net/starlingx/+bug/1821938


From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: Wednesday, April 10, 2019 11:12 PM
To: Khalil, Ghada; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io
Cc: Liu, Yang
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

Hi Ghada,

Logs on nova pod are not showing any ERROR, just this warning (log shows that is fixed on the next line):

2019-04-10 12:52:47,173.173 229781 WARNING nova.conductor.api [req-c557fabd-b379-4dda-b64e-981e71b530bf - - - - -] Timed out waiting for nova-conductor.  Is it running? Or did this service start before nova-conductor?  Reattempting establishment of nova-conductor connection...: MessagingTimeout: Timed out waiting for a reply to message ID af6fd23a7b6a476985949f42947f0d4e
2019-04-10 12:52:47,192.192 229781 INFO nova.conductor.api [req-c557fabd-b379-4dda-b64e-981e71b530bf - - - - -] nova-conductor connection established successfully

I’m not sure about the output of  nova hypervisor-list

DEBUG (shell:812) internalURL endpoint for compute service in RegionOne region not found
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 810, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 678, in main
    api_version = api_versions.discover_version(self.cs, api_version)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 261, in discover_version
    client)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 242, in _get_server_version_range
    version = client.versions.get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 70, in get_current
    return self._get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 52, in _get_current
    url = "%s" % self.api.client.get_endpoint()
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 271, in get_endpoint
    return self.session.get_endpoint(auth or self.auth, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1139, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 279, in get_endpoint_data
    service_name=service_name)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
    raise exceptions.EndpointNotFound(msg)
EndpointNotFound: internalURL endpoint for compute service in RegionOne region not found

Regards,

Cristopher Lemus


From: "Khalil, Ghada" <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Wednesday, April 10, 2019 at 9:50 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com<mailto:cristopher.j.lemus.contreras at intel.com>>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra at intel.com<mailto:maria.g.perez.ibarra at intel.com>>, "starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: "Liu, Yang" <yang.liu at windriver.com<mailto:yang.liu at windriver.com>>
Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

Hi Christopher,
These images look correct to me, but I’ll ask Yang to confirm the images she tested with tomorrow.

Can you confirm that nova is still returning the same error as was originally reported in https://bugs.launchpad.net/starlingx/+bug/1823275

nova is returning this error:
2019-04-04 16:22:30,902.902 168396 ERROR nova.compute.manager [req-b3bffaba-a62a-4e56-be63-3362c51a36df - - - - -] Error updating resources for node compute-0.: PciDeviceNotFoundById: PCI device 0000:b3:02.3 not found

Can you also list the output of the nova hypervisor cmd on the compute node?
nova hypervisor-list

Thanks,
Ghada

From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: Wednesday, April 10, 2019 10:40 PM
To: Khalil, Ghada; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Cc: Liu, Yang
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

Hi Ghada,

These are the starlingx nova images used on the servers where the sanity was executed:

controller-0:~# docker images |grep starlingx/stx-nova
192.168.100.60/starlingx/stx-nova                                                   master-centos-stable-latest                6c8a3a356110        30 hours ago        1.18GB
192.168.204.2:9001/docker.io/starlingx/stx-nova                                     master-centos-stable-latest                6c8a3a356110        30 hours ago        1.18GB
192.168.100.60/starlingx/stx-nova-api-proxy                                         master-centos-stable-latest                c5853883d561        31 hours ago        589MB
192.168.204.2:9001/docker.io/starlingx/stx-nova-api-proxy                           master-centos-stable-latest                c5853883d561        31 hours ago        589MB
controller-0:~#


For stx-nova, id: 6c8a3a356110           and for stx-nova-api-proxy, id: c5853883d561


I manually verified the images on our local registry, and tried to download the latest version:

[root at registry ~]# docker pull starlingx/stx-nova:master-centos-stable-latest
master-centos-stable-latest: Pulling from starlingx/stx-nova
Digest: sha256:e9fecb6998ad0cf0a6621a8c5c2422378d0972c73ed80b636dcbd4bb5183794e
Status: Image is up to date for starlingx/stx-nova:master-centos-stable-latest
[root at registry ~]# docker pull starlingx/stx-nova-api-proxy:master-centos-stable-latest
master-centos-stable-latest: Pulling from starlingx/stx-nova-api-proxy
Digest: sha256:06c9bf3546eeec5025e62ccf00c24dbaa20e6d79614d89709c0a71ffa639c5f8
Status: Image is up to date for starlingx/stx-nova-api-proxy:master-centos-stable-latest
[root at registry ~]# docker images |grep starlingx/stx-nova
registry.zpn.intel.com/starlingx/stx-nova                                               master-centos-stable-latest                6c8a3a356110        28 hours ago        1.18GB
starlingx/stx-nova                                                                      master-centos-stable-latest                6c8a3a356110        28 hours ago        1.18GB
starlingx/stx-nova-api-proxy                                                            master-centos-stable-latest                c5853883d561        29 hours ago        589MB
registry.zpn.intel.com/starlingx/stx-nova-api-proxy                                     master-centos-stable-latest                c5853883d561        29 hours ago        589MB

Images are up to date, and the timestamp matches what is logged in dockerhub: https://hub.docker.com/r/starlingx/stx-nova/tags . I assume that those are the images according to the comments on https://bugs.launchpad.net/starlingx/+bug/1821938, are there other nova images that we need to pull? I did a quick check and it looks like all of our local images (the ones deployed to the sanity servers) are updated.

Thanks & Regards,

Cristopher Lemus




From: "Khalil, Ghada" <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Wednesday, April 10, 2019 at 8:27 PM
To: "Perez Ibarra, Maria G" <maria.g.perez.ibarra at intel.com<mailto:maria.g.perez.ibarra at intel.com>>, "starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: "Liu, Yang" <yang.liu at windriver.com<mailto:yang.liu at windriver.com>>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410

Hi Maria,
Regarding https://bugs.launchpad.net/starlingx/+bug/1821938, Yang successfully verified that the issue is resolved using the same load you list below (see her note in the LP).
Can you confirm that you are using the stable docker images built yesterday? The fix should be included in the nova image.

Thanks,
Ghada

From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra at intel.com]
Sent: Wednesday, April 10, 2019 8:59 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410


Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-10 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190410T013000Z/outputs/iso/>)



Status: YELLOW



===========================================



Sanity Test is executed in a Containers – Bare Metal Environment



AIO - Simplex



Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  49 TCs [FAIL] | 10 TCs FAIL

Sanity Platform   07 TCs [FAIL] | 03 TCs FAIL



TOTAL: 57 TCS [PASS : 44] [Fail : 13]


AIO – Duplex



Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] | 10 TCs FAIL

Sanity Platform   05 TCs [PASS] | 01 TCs FAIL



TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 TCs]



Standard - Local Storage (2+2)



Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] | 10 TCs FAIL

Sanity Platform   05 TCs [PASS] | 01 TCs FAIL



TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 TCs]



Standard - Dedicated Storage (2+2+2)



Setup             Manual [PASS]

Provisioning      01 TCs [PASS]

Sanity OpenStack  52 TCs [PASS] | 10 TCs FAIL

Sanity Platform   05 TCs [PASS] | 01 TCs FAIL



TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 Tcs]





Sanity Test is executed in a Containers - Virtual Environment



AIO - Simplex


Setup             04 TCs [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  49 TCs [PASS]
Sanity Platform   07 TCs [PASS]

TOTAL: [ 61 TCs PASS ]



AIO - Duplex


Setup             04 TCs [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  49 TCs [PASS]
Sanity Platform   07 TCs [PASS]

TOTAL: [ 61 TCs PASS ]


Standard – Local Storage

Setup             04 TCs [PASS]
Provisioning      01 TCs [PASS]
Sanity OpenStack  52 TCs [PASS]
Sanity Platform   05 TCs [PASS]

TOTAL: [ 61 TCs PASS ]



--------------------------------------------------------------------------------
No nova hypervisor can be enabled on workers with QAT devices  https://bugs.launchpad.net/starlingx/+bug/1821938

For more detail of the tests: https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack


Regards
Maria G.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190411/19bd4c7f/attachment-0001.html>


More information about the Starlingx-discuss mailing list