Hi Nic & All,

We(Me, Sanjay & Sharath) have taken image from link you shared below and still facing same issue.

I am not sure about type of error (it was working fine until 2 days back), Just a guess is it anywhere related to local Proxy setting ? Any input/suggestions would help.

[sysadmin@controller-0 ~(keystone_admin)]$ tail -f /var/log/sysinv.log

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp app_applied = self._app.perform_app_apply(rpc_app, mode, lifecycle_hook_info_app_apply)

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1995, in perform_app_apply

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application stx-openstack (1.0-49-centos-stable-versioned) failed: (404)

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp Reason: Not Found

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 05 Feb 2021 13:40:32 GMT', 'Content-Length': '210', 'Content-Type': 'application/json'})

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"ceph-pool-kube-rbd\" not found","reason":"NotFound","details":{"name":"ceph-pool-kube-rbd

","kind":"secrets"},"code":404}

2021-02-05 13:40:32.592 93081 ERROR sysinv.openstack.common.rpc.amqp

Regards,

Haridhar Kalvala

From: Jascanu, Nicolae <nicolae.jascanu@intel.com>
Sent: Thursday, February 4, 2021 8:22 PM
To: Kalvala, Haridhar <haridhar.kalvala@intel.com>; starlingx-discuss@lists.starlingx.io
Cc: Kumar, Sharath <sharath.kumar@intel.com>; Mukherjee, Sanjay K <sanjay.k.mukherjee@intel.com>
Subject: RE: Default Helm-chart apply failing (4.0.1)

Hi All,

We installed the 4.0.1 from (http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/flock/outputs/) on a Virtual Standard configuration in two scenarios: downloading the images from internet and using our mirror registry that already has the images.

We managed to fully install and execute sanity on both cases.

We are not able to replicate your issue.

Regards,

Nicolae Jascanu, Ph.D.

Software Engineer

IOTG

Galati, Romania

From: Kalvala, Haridhar <haridhar.kalvala@intel.com>
Sent: Thursday, February 4, 2021 05:16
To: starlingx-discuss@lists.starlingx.io
Cc: Kumar, Sharath <sharath.kumar@intel.com>
Subject: Re: [Starlingx-discuss] Default Helm-chart apply failing (4.0.1)

Hello All,

Any update/comment on below issue. It still persists with any version stx-openstack app(Ex: stx-openstack-1.0-49-centos-stable-versioned.tgz) ?

Thank you,

Haridhar Kalvala

Pasting error log again :

ksysadmin@controller-0 ~(keystone_admin)]$ system application-apply stx-openstack

+---------------+----------------------------------+

| Property | Value |

+---------------+----------------------------------+

| active | False |

| app_version | 1.0-49-centos-stable-versioned |

| created_at | 2021-02-04T11:08:55.408108+00:00 |

| manifest_file | stx-openstack.yaml |

| manifest_name | armada-manifest |

| name | stx-openstack |

| progress | None |

| status | applying |

| updated_at | 2021-02-04T11:09:20.815085+00:00 |

+---------------+----------------------------------+

Please use 'system application-list' or 'system application-show stx-openstack' to view the current progress.

[sysadmin@controller-0 ~(keystone_admin)]$ tail -f /var/log/sysinv.log

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp app_applied = self._app.perform_app_apply(rpc_app, mode, lifecycle_hook_info_app_apply)

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1995, in perform_app_apply

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application stx-openstack (1.0-49-centos-stable-versioned) failed: (404)

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp Reason: Not Found

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 04 Feb 2021 11:09:38 GMT', 'Content-Length': '210', 'Content-Type': 'application/json'})

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"ceph-pool-kube-rbd\" not found","reason":"NotFound","details":{"name":"ceph-pool-kube-rbd","kind":"secrets"},"code":404}

2021-02-04 11:09:39.231 99042 ERROR sysinv.openstack.common.rpc.amqp

sysinv 2021-02-04 11:10:20.070 99042 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://localhost:30001/nfvi-plugins/v1/sw-update hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:None

sysinv 2021-02-04 11:10:20.071 99042 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'success', u'in-progress': None, u'sw-update-type': None}

sysinv 2021-02-04 11:10:20.117 99042 INFO sysinv.conductor.kube_app [-] lifecycle hook for application platform-integ-apps (1.0-9) started {'lifecycle_type': u'check', 'relative_timing': u'pre', 'mode': u'auto', 'operation': u'apply', 'extra': {}}.

sysinv 2021-02-04 11:10:20.117 99042 INFO sysinv.conductor.manager [-] Auto-apply failed prerequisites for platform-integ-apps: Automatic apply is disabled for platform-integ-apps.

sysinv 2021-02-04 11:10:20.122 99042 INFO sysinv.conductor.kube_app [-] lifecycle hook for application oidc-auth-apps (1.0-29) started {'lifecycle_type': u'check', 'relative_timing': u'pre', 'mode': u'auto', 'operation': u'apply', 'extra': {}}.

sysinv 2021-02-04 11:10:20.123 99042 INFO sysinv.conductor.manager [-] Auto-apply failed prerequisites for oidc-auth-apps: Automatic apply is disabled for oidc-auth-apps.

sysinv 2021-02-04 11:10:20.131 99042 INFO sysinv.api.controllers.v1.rest_api [-] GET cmd:http://localhost:30001/nfvi-plugins/v1/sw-update hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:None

sysinv 2021-02-04 11:10:20.132 99042 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'status': u'success', u'in-progress': None, u'sw-update-type': None}

From: Kalvala, Haridhar
Sent: Wednesday, February 3, 2021 4:39 PM
To: Kumar, Sharath <sharath.kumar@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: Default Helm-chart apply failing (4.0.1)

Even I am observing the same since morning (IST time zone).

Error log:

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp app_applied = self._app.perform_app_apply(rpc_app, mode, lifecycle_hook_info_app_apply)

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1995, in perform_app_apply

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application stx-openstack (1.0-49-centos-stable-versioned) failed: (404)

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp Reason: Not Found

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 03 Feb 2021 18:37:19 GMT', 'Content-Length': '210', 'Content-Type': 'application/json'})

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"ceph-pool-kube-rbd\" not found","reason":"NotFound","details":{"name":"ceph-pool-kube-rbd

","kind":"secrets"},"code":404}

2021-02-03 18:37:19.530 99042 ERROR sysinv.openstack.common.rpc.amqp

Regards,

Haridhar Kalvala

From: Kumar, Sharath <sharath.kumar@intel.com>
Sent: Wednesday, February 3, 2021 4:14 PM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] Default Helm-chart apply failing (4.0.1)

Hi All,

We are seeing stx-openstack Armanda helm chart getting failed during apply. Initial docker images are not downloading and throwing error. It was working till last night and since morning am seeing this issue. Kindly fix the issue and do the needful. Thank you in advance.

Regards,

Sharath