[Starlingx-discuss] Default Helm-chart apply failing (4.0.1)

Fri Feb 5 05:48:42 UTC 2021

Hi Nic & All,

We(Me, Sanjay & Sharath) have taken image from link you shared below and still facing same issue.
I am not sure about type of error (it was working fine until 2 days back), Just a guess is it anywhere related to local Proxy setting ? Any input/suggestions would help.

[sysadmin at controller-0 ~(keystone_admin)]$ tail -f /var/log/sysinv.log
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp     app_applied = self._app.perform_app_apply(rpc_app, mode, lifecycle_hook_info_app_apply)
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1995, in perform_app_apply
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp     name=app.name, version=app.version, reason=e)
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application stx-openstack (1.0-49-centos-stable-versioned) failed: (404)
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp Reason: Not Found
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 05 Feb 2021 13:40:32 GMT', 'Content-Length': '210', 'Content-Type': 'application/json'})
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"ceph-pool-kube-rbd\" not found","reason":"NotFound","details":{"name":"ceph-pool-kube-rbd
","kind":"secrets"},"code":404}
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp
2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp

Regards,
Haridhar Kalvala

From: Jascanu, Nicolae <nicolae.jascanu at intel.com>
Sent: Thursday, February 4, 2021 8:22 PM
To: Kalvala, Haridhar <haridhar.kalvala at intel.com>; starlingx-discuss at lists.starlingx.io
Cc: Kumar, Sharath <sharath.kumar at intel.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee at intel.com>
Subject: RE: Default Helm-chart apply failing (4.0.1)

Hi All,
We installed the 4.0.1 from (http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/flock/outputs/) on a Virtual Standard configuration in two scenarios: downloading the images from internet and using our mirror registry that already has the images.
We managed to fully install and execute sanity on both cases.
We are not able to replicate your issue.

Regards,
Nicolae Jascanu, Ph.D.
Software Engineer
IOTG
Galati, Romania

From: Kalvala, Haridhar <haridhar.kalvala at intel.com<mailto:haridhar.kalvala at intel.com>>
Sent: Thursday, February 4, 2021 05:16
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Cc: Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>
Subject: Re: [Starlingx-discuss] Default Helm-chart apply failing (4.0.1)

Hello All,

Any update/comment on below issue. It still persists with any version stx-openstack app(Ex: stx-openstack-1.0-49-centos-stable-versioned.tgz) ?

Thank you,
Haridhar Kalvala

Pasting error log again :

ksysadmin at controller-0 ~(keystone_admin)]$ system application-apply stx-openstack
+---------------+----------------------------------+
| Property      | Value                            |
+---------------+----------------------------------+
| active        | False                            |
| app_version   | 1.0-49-centos-stable-versioned   |
| created_at    | 2021-02-04T11:08:55.408108+00:00 |
| manifest_file | stx-openstack.yaml               |
| manifest_name | armada-manifest                  |
| name          | stx-openstack                    |
| progress      | None                             |
| status        | applying                         |
| updated_at    | 2021-02-04T11:09:20.815085+00:00 |
+---------------+----------------------------------+
Please use 'system application-list' or 'system application-show stx-openstack' to view the current progress.
[sysadmin at controller-0 ~(keystone_admin)]$ tail -f /var/log/sysinv.log
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp     app_applied = self._app.perform_app_apply(rpc_app, mode, lifecycle_hook_info_app_apply)
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1995, in perform_app_apply
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp     name=app.name, version=app.version, reason=e)
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application stx-openstack (1.0-49-centos-stable-versioned) failed: (404)
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp Reason: Not Found
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 04 Feb 2021 11:09:38 GMT', 'Content-Length': '210', 'Content-Type': 'application/json'})
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"ceph-pool-kube-rbd\" not found","reason":"NotFound","details":{"name":"ceph-pool-kube-rbd","kind":"secrets"},"code":404}
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp
2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp
sysinv 2021-02-04 11:10:20.070 99042 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://localhost:30001/nfvi-plugins/v1/sw-update hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:None
sysinv 2021-02-04 11:10:20.071 99042 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'success', u'in-progress': None, u'sw-update-type': None}
sysinv 2021-02-04 11:10:20.117 99042 INFO sysinv.conductor.kube_app [-] lifecycle hook for application platform-integ-apps (1.0-9) started {'lifecycle_type': u'check', 'relative_timing': u'pre', 'mode': u'auto', 'operation': u'apply', 'extra': {}}.
sysinv 2021-02-04 11:10:20.117 99042 INFO sysinv.conductor.manager [-] Auto-apply failed prerequisites for platform-integ-apps: Automatic apply is disabled for platform-integ-apps.
sysinv 2021-02-04 11:10:20.122 99042 INFO sysinv.conductor.kube_app [-] lifecycle hook for application oidc-auth-apps (1.0-29) started {'lifecycle_type': u'check', 'relative_timing': u'pre', 'mode': u'auto', 'operation': u'apply', 'extra': {}}.
sysinv 2021-02-04 11:10:20.123 99042 INFO sysinv.conductor.manager [-] Auto-apply failed prerequisites for oidc-auth-apps: Automatic apply is disabled for oidc-auth-apps.
sysinv 2021-02-04 11:10:20.131 99042 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://localhost:30001/nfvi-plugins/v1/sw-update hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:None
sysinv 2021-02-04 11:10:20.132 99042 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'success', u'in-progress': None, u'sw-update-type': None}

From: Kalvala, Haridhar
Sent: Wednesday, February 3, 2021 4:39 PM
To: Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: RE: Default Helm-chart apply failing (4.0.1)

Even I am observing the same since morning (IST time zone).

Error log:

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp     app_applied = self._app.perform_app_apply(rpc_app, mode, lifecycle_hook_info_app_apply)
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1995, in perform_app_apply
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp     name=app.name, version=app.version, reason=e)
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application stx-openstack (1.0-49-centos-stable-versioned) failed: (404)
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp Reason: Not Found
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 03 Feb 2021 18:37:19 GMT', 'Content-Length': '210', 'Content-Type': 'application/json'})
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"ceph-pool-kube-rbd\" not found","reason":"NotFound","details":{"name":"ceph-pool-kube-rbd
","kind":"secrets"},"code":404}
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp
2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp

Regards,
Haridhar Kalvala

From: Kumar, Sharath <sharath.kumar at intel.com<mailto:sharath.kumar at intel.com>>
Sent: Wednesday, February 3, 2021 4:14 PM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] Default Helm-chart apply failing (4.0.1)

Hi All,

We are seeing stx-openstack Armanda helm chart getting failed during apply. Initial docker images are not downloading and throwing error. It was working till last night and since morning am seeing this issue. Kindly fix the issue and do the needful. Thank you in advance.

Regards,
Sharath
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210205/7f9c7e4d/attachment-0001.html>