[Starlingx-discuss] Enable PCI pass-through failed

Sat Nov 6 06:03:38 UTC 2021

Hi Thiago,
  I reinstalled the latest_green_build (from Oct 27th) successfully, and try to override with PCI passthrough using the "nova-tsn-pt.yaml". I still get the ceph nil pointer error during applying:

2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller [-] [chart=openstack-nova]: Error while updating release osh-openstack-nova: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
  status = StatusCode.UNKNOWN
  details = "render error in "nova/job-storage-init.yaml": template: nova/job-storage-init.yaml:65:24: executing "nova/job-storage-init.yaml" at <.Values.conf.ceph.enabled>: nil pointer evaluating interface {}.ceph"
  debug_error_string = "{"created":"@1636177646.883937903","description":"Error received from peer ipv4:127.0.0.1:24134","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"render error in "nova/job-storage-init.yaml": template: nova/job-storage-init.yaml:65:24: executing "nova/job-storage-init.yaml" at <.Values.conf.ceph.enabled>: nil pointer evaluating interface {}.ceph","grpc_status":2}"

2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller Traceback (most recent call last):
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 421, in update_release
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller     metadata=self.metadata)
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 923, in __call__
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller     return _end_unary_response_blocking(state, call, False, None)
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller     raise _InactiveRpcError(state)
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller  status = StatusCode.UNKNOWN
2021-11-06 05:47:26.884 398 ERROR armada.handlers.tiller  details = "render error in "nova/job-storage-init.yaml": template: nova/job-storage-init.yaml:65:24: executing "nova/job-storage-init.yaml" at <.Values.conf.ceph.enabled>: nil pointer evaluating interface {}.ceph"

I cannot attach the log file due to "Message body is too big". Hope this can be solved soon! Thank you!

Regards,
Sinlam

________________________________
From: Brito, Thiago <Thiago.Brito at windriver.com>
Sent: Friday, November 5, 2021 9:18 PM
To: Sin Lam Tan <sinlam at hotmail.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: Re: Enable PCI pass-through failed

Hi Sinlam,

I'd have to have more information about why that pod didn't get up. Can you get me the logs and the describe output for that?

Thiago
________________________________
From: Sin Lam Tan <sinlam at hotmail.com>
Sent: Friday, November 5, 2021 3:34 AM
To: Brito, Thiago <Thiago.Brito at windriver.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: Re: Enable PCI pass-through failed

[Please note: This e-mail is from an EXTERNAL e-mail address]

Hi Thiago,
   Thanks for the suggestion. I tried the latest_green_build (from Oct 27th) that you listed below. I haven't reached the stx-openstack override yet, just did the normal Openstack installation step and it failed (I tried it twice). The apply failed in osh-openstack-fm-rest-api (timeout at 59%) during apply of stx-openstack-1.0-119-centos-stable-latest. Below is the detailed log:

2021-11-05 06:22:30.431 320 ERROR armada.handlers.armada armada.exceptions.k8s_exceptions.KubernetesWatchTimeoutException: Timed out waiting for pods (namespace=openstack, labels=(release_group=osh-openstack-fm-rest-api)). These pods were not ready=['fm-rest-api-75ccb49d86-vlg8h']

Previous build (stx-openstack-1.0-116) installed ok for Openstack, which I tried yesterday. Any suggestion to solve this new issue?

Regards,
Sinlam

________________________________
From: Brito, Thiago <Thiago.Brito at windriver.com>
Sent: Thursday, November 4, 2021 10:06 PM
To: Sin Lam Tan <sinlam at hotmail.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: Re: Enable PCI pass-through failed

Good day Sinlam,

We are very close on taking that issue down, but you might want to try the latest_green_build (from Oct 27th): http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green_build/<https://urldefense.com/v3/__http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green_build/__;!!AjveYdw8EvQ!Mcqxx-XTEtOEbneXgRvSCVUeLR9NbKwpT1lAiR4orxgEWUduKKTU6vHpxHUkNQiWloNl$>

Please, give it a try and let me know.

Thiago
________________________________
From: Sin Lam Tan <sinlam at hotmail.com>
Sent: Wednesday, November 3, 2021 8:35 PM
To: Brito, Thiago <Thiago.Brito at windriver.com>
Subject: Re: Enable PCI pass-through failed

[Please note: This e-mail is from an EXTERNAL e-mail address]

Hi Thiago,
  Thanks for your response. Previously I did application applied w/o errors before these overrides.
Is there any older version of Starlingx that I can try without this ceph upversion issue?
Thank you!

Regards,
Sinlam

________________________________
From: Brito, Thiago <Thiago.Brito at windriver.com>
Sent: Thursday, November 4, 2021 2:23 AM
To: Sin Lam Tan <sinlam at hotmail.com>
Subject: Re: Enable PCI pass-through failed

Hi Sinlam,

We are facing some issues on stx-openstack related to the ceph upversion that might be related to what you saw (not exactly related to what you are trying to do).

Was the application applied w/o errors before you tried these overrides?

Thiago
________________________________
From: Sin Lam Tan <sinlam at hotmail.com>
Sent: Monday, November 1, 2021 6:25 AM
To: starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Enable PCI pass-through failed

[Please note: This e-mail is from an EXTERNAL e-mail address]

Dear all,
  I followed the documentation at https://wiki.openstack.org/wiki/StarlingX/Networking/TSN<https://urldefense.com/v3/__https://wiki.openstack.org/wiki/StarlingX/Networking/TSN__;!!AjveYdw8EvQ!KqGY8m7iL-FK6KU17XYrMCddkEnQjSzOe28b9dVxpPk8WupQcUHlXJ5EmbSMAPyvrYla$> to enable PCI passthrough so that Starlingx can support TSN. In Starlingx AIO host, I have i210 network interface. In the documentation, it mentioned about creating "nova-tsn-pt.yaml".

1/ Inside this file, is the "{host_ip}" point to my OAM IP address?

2/ The "{controller_name}" can I replace with the default name "controller-0"?

When I issue "system helm-override-update stx-openstack nova openstack --values nova-tsn-pt.yaml" and "system application-apply stx-openstack", it failed with the error below:

2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller [-] [chart=openstack-nova]: Error while updating release osh-openstack-nova: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "render error in "nova/job-storage-init.yaml": template: nova/job-storage-init.yaml:65:24: executing "nova/job-storage-init.yaml" at <.Values.conf.ceph.enabled>: nil pointer evaluating interface {}.ceph"
debug_error_string = "{"created":"@1635757332.936926142","description":"Error received from peer ipv4:127.0.0.1:24134","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"render error in "nova/job-storage-init.yaml": template: nova/job-storage-init.yaml:65:24: executing "nova/job-storage-init.yaml" at <.Values.conf.ceph.enabled>: nil pointer evaluating interface {}.ceph","grpc_status":2}"
>
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller Traceback (most recent call last):
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/armada/handlers/tiller.py", line 421, in update_release
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller     metadata=self.metadata)
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 923, in __call__
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller     return _end_unary_response_blocking(state, call, False, None)
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller   File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller     raise _InactiveRpcError(state)
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller status = StatusCode.UNKNOWN
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller details = "render error in "nova/job-storage-init.yaml": template: nova/job-storage-init.yaml:65:24: executing "nova/job-storage-init.yaml" at <.Values.conf.ceph.enabled>: nil pointer evaluating interface {}.ceph"
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller debug_error_string = "{"created":"@1635757332.936926142","description":"Error received from peer ipv4:127.0.0.1:24134","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"render error in "nova/job-storage-init.yaml": template: nova/job-storage-init.yaml:65:24: executing "nova/job-storage-init.yaml" at <.Values.conf.ceph.enabled>: nil pointer evaluating interface {}.ceph","grpc_status":2}"
2021-11-01 09:02:12.937 144 ERROR armada.handlers.tiller >

How can I solve the above error?

I am using the latest Starlingx V5.0.1 with the Openstack "stx-openstack-1.0-116-centos-stable-latest.tgz". Thanks for any information!

Regards,
Sinlam

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20211106/5c35f349/attachment-0001.htm>