[Starlingx-discuss] [Containers] Sanity Test - ISO 20190410
Liu, Yang
yang.liu at windriver.com
Thu Apr 11 18:39:48 UTC 2019
Are you able to run any nova cmd Christopher?
$ source /etc/platform/openrc ; system host-device-list compute-0; export OS_AUTH_URL=http://keystone.openstack.svc.cluster.local/v3; nova hypervisor-list
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
| name | address | class id | vendor id | device id | class name | vendor name | device name | numa_node | enabled |
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
| pci_0000_08_00_0 | 0000:08:00.0 | 0b4000 | 8086 | 0435 | Co-processor | Intel Corporation | DH895XCC Series QAT | 0 | True |
| pci_0000_0c_00_0 | 0000:0c:00.0 | 030000 | 102b | 0522 | VGA compatible controller | Matrox Electronics Systems Ltd. | MGA G200e [Pilot] ServerEngines (SEP1) | 0 | True |
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
+--------------------------------------+---------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+---------------------+-------+---------+
| 6136b80d-e2a9-4d34-97ce-d3818b305a7f | compute-0 | up | enabled |
| d87821b4-4575-4238-b97c-b07589465127 | compute-2 | up | enabled |
| f620e8ae-7353-4fbb-ad0d-f427b6c8bd00 | compute-3 | up | enabled |
| a966bf58-6ce8-42d4-be8b-ea4453305644 | compute-1 | up | enabled |
+--------------------------------------+---------------------+-------+---------+
Images are the same btw.
192.168.204.2:9001/docker.io/starlingx/stx-nova master-centos-stable-latest 6c8a3a356110 2 days ago 1.18GB
starlingx/stx-nova master-centos-stable-latest 6c8a3a356110 2 days ago 1.18GB
192.168.204.2:9001/docker.io/starlingx/stx-nova-api-proxy master-centos-stable-latest c5853883d561 2 days ago 589MB
starlingx/stx-nova-api-proxy master-centos-stable-latest c5853883d561 2 days ago 589MB
BR,
Yang
From: Khalil, Ghada
Sent: April-11-19 2:10 PM
To: Lemus Contreras, Cristopher J; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io
Cc: Liu, Yang
Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410
This looks like a different issue unrelated to the original issue reported in
https://bugs.launchpad.net/starlingx/+bug/1823275
https://bugs.launchpad.net/starlingx/+bug/1821938
From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: Wednesday, April 10, 2019 11:12 PM
To: Khalil, Ghada; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io
Cc: Liu, Yang
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410
Hi Ghada,
Logs on nova pod are not showing any ERROR, just this warning (log shows that is fixed on the next line):
2019-04-10 12:52:47,173.173 229781 WARNING nova.conductor.api [req-c557fabd-b379-4dda-b64e-981e71b530bf - - - - -] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? Reattempting establishment of nova-conductor connection...: MessagingTimeout: Timed out waiting for a reply to message ID af6fd23a7b6a476985949f42947f0d4e
2019-04-10 12:52:47,192.192 229781 INFO nova.conductor.api [req-c557fabd-b379-4dda-b64e-981e71b530bf - - - - -] nova-conductor connection established successfully
I’m not sure about the output of nova hypervisor-list
DEBUG (shell:812) internalURL endpoint for compute service in RegionOne region not found
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 810, in main
OpenStackComputeShell().main(argv)
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 678, in main
api_version = api_versions.discover_version(self.cs, api_version)
File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 261, in discover_version
client)
File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 242, in _get_server_version_range
version = client.versions.get_current()
File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 70, in get_current
return self._get_current()
File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 52, in _get_current
url = "%s" % self.api.client.get_endpoint()
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 271, in get_endpoint
return self.session.get_endpoint(auth or self.auth, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1139, in get_endpoint
return auth.get_endpoint(self, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
allow_version_hack=allow_version_hack, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 279, in get_endpoint_data
service_name=service_name)
File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for
raise exceptions.EndpointNotFound(msg)
EndpointNotFound: internalURL endpoint for compute service in RegionOne region not found
Regards,
Cristopher Lemus
From: "Khalil, Ghada" <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Wednesday, April 10, 2019 at 9:50 PM
To: "Lemus Contreras, Cristopher J" <cristopher.j.lemus.contreras at intel.com<mailto:cristopher.j.lemus.contreras at intel.com>>, "Perez Ibarra, Maria G" <maria.g.perez.ibarra at intel.com<mailto:maria.g.perez.ibarra at intel.com>>, "starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: "Liu, Yang" <yang.liu at windriver.com<mailto:yang.liu at windriver.com>>
Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410
Hi Christopher,
These images look correct to me, but I’ll ask Yang to confirm the images she tested with tomorrow.
Can you confirm that nova is still returning the same error as was originally reported in https://bugs.launchpad.net/starlingx/+bug/1823275
nova is returning this error:
2019-04-04 16:22:30,902.902 168396 ERROR nova.compute.manager [req-b3bffaba-a62a-4e56-be63-3362c51a36df - - - - -] Error updating resources for node compute-0.: PciDeviceNotFoundById: PCI device 0000:b3:02.3 not found
Can you also list the output of the nova hypervisor cmd on the compute node?
nova hypervisor-list
Thanks,
Ghada
From: Lemus Contreras, Cristopher J [mailto:cristopher.j.lemus.contreras at intel.com]
Sent: Wednesday, April 10, 2019 10:40 PM
To: Khalil, Ghada; Perez Ibarra, Maria G; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Cc: Liu, Yang
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410
Hi Ghada,
These are the starlingx nova images used on the servers where the sanity was executed:
controller-0:~# docker images |grep starlingx/stx-nova
192.168.100.60/starlingx/stx-nova master-centos-stable-latest 6c8a3a356110 30 hours ago 1.18GB
192.168.204.2:9001/docker.io/starlingx/stx-nova master-centos-stable-latest 6c8a3a356110 30 hours ago 1.18GB
192.168.100.60/starlingx/stx-nova-api-proxy master-centos-stable-latest c5853883d561 31 hours ago 589MB
192.168.204.2:9001/docker.io/starlingx/stx-nova-api-proxy master-centos-stable-latest c5853883d561 31 hours ago 589MB
controller-0:~#
For stx-nova, id: 6c8a3a356110 and for stx-nova-api-proxy, id: c5853883d561
I manually verified the images on our local registry, and tried to download the latest version:
[root at registry ~]# docker pull starlingx/stx-nova:master-centos-stable-latest
master-centos-stable-latest: Pulling from starlingx/stx-nova
Digest: sha256:e9fecb6998ad0cf0a6621a8c5c2422378d0972c73ed80b636dcbd4bb5183794e
Status: Image is up to date for starlingx/stx-nova:master-centos-stable-latest
[root at registry ~]# docker pull starlingx/stx-nova-api-proxy:master-centos-stable-latest
master-centos-stable-latest: Pulling from starlingx/stx-nova-api-proxy
Digest: sha256:06c9bf3546eeec5025e62ccf00c24dbaa20e6d79614d89709c0a71ffa639c5f8
Status: Image is up to date for starlingx/stx-nova-api-proxy:master-centos-stable-latest
[root at registry ~]# docker images |grep starlingx/stx-nova
registry.zpn.intel.com/starlingx/stx-nova master-centos-stable-latest 6c8a3a356110 28 hours ago 1.18GB
starlingx/stx-nova master-centos-stable-latest 6c8a3a356110 28 hours ago 1.18GB
starlingx/stx-nova-api-proxy master-centos-stable-latest c5853883d561 29 hours ago 589MB
registry.zpn.intel.com/starlingx/stx-nova-api-proxy master-centos-stable-latest c5853883d561 29 hours ago 589MB
Images are up to date, and the timestamp matches what is logged in dockerhub: https://hub.docker.com/r/starlingx/stx-nova/tags . I assume that those are the images according to the comments on https://bugs.launchpad.net/starlingx/+bug/1821938, are there other nova images that we need to pull? I did a quick check and it looks like all of our local images (the ones deployed to the sanity servers) are updated.
Thanks & Regards,
Cristopher Lemus
From: "Khalil, Ghada" <Ghada.Khalil at windriver.com<mailto:Ghada.Khalil at windriver.com>>
Date: Wednesday, April 10, 2019 at 8:27 PM
To: "Perez Ibarra, Maria G" <maria.g.perez.ibarra at intel.com<mailto:maria.g.perez.ibarra at intel.com>>, "starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>" <starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>>
Cc: "Liu, Yang" <yang.liu at windriver.com<mailto:yang.liu at windriver.com>>
Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410
Hi Maria,
Regarding https://bugs.launchpad.net/starlingx/+bug/1821938, Yang successfully verified that the issue is resolved using the same load you list below (see her note in the LP).
Can you confirm that you are using the stable docker images built yesterday? The fix should be included in the nova image.
Thanks,
Ghada
From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra at intel.com]
Sent: Wednesday, April 10, 2019 8:59 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190410
Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-10 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190410T013000Z/outputs/iso/>)
Status: YELLOW
===========================================
Sanity Test is executed in a Containers – Bare Metal Environment
AIO - Simplex
Setup Manual [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [FAIL] | 10 TCs FAIL
Sanity Platform 07 TCs [FAIL] | 03 TCs FAIL
TOTAL: 57 TCS [PASS : 44] [Fail : 13]
AIO – Duplex
Setup Manual [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS] | 10 TCs FAIL
Sanity Platform 05 TCs [PASS] | 01 TCs FAIL
TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 TCs]
Standard - Local Storage (2+2)
Setup Manual [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS] | 10 TCs FAIL
Sanity Platform 05 TCs [PASS] | 01 TCs FAIL
TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 TCs]
Standard - Dedicated Storage (2+2+2)
Setup Manual [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS] | 10 TCs FAIL
Sanity Platform 05 TCs [PASS] | 01 TCs FAIL
TOTAL: 57 TCS [PASS : 46 TCs] [Fail : 11 Tcs]
Sanity Test is executed in a Containers - Virtual Environment
AIO - Simplex
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
AIO - Duplex
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 49 TCs [PASS]
Sanity Platform 07 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
Standard – Local Storage
Setup 04 TCs [PASS]
Provisioning 01 TCs [PASS]
Sanity OpenStack 52 TCs [PASS]
Sanity Platform 05 TCs [PASS]
TOTAL: [ 61 TCs PASS ]
--------------------------------------------------------------------------------
No nova hypervisor can be enabled on workers with QAT devices https://bugs.launchpad.net/starlingx/+bug/1821938
For more detail of the tests: https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack
Regards
Maria G.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190411/19bd4c7f/attachment-0001.html>
More information about the Starlingx-discuss
mailing list