There is an existing LP with the same cause: https://bugs.launchpad.net/starlingx/+bug/1833323

Until it is resolved, issue the following commands if the application status are stuck in applying/removing because sysinv-conductor was abruptly terminated (power outage, process restarted by sm due to missing audits, process killed/system boot due to OOM, etc…)

sudo -u postgres psql -d sysinv -c "update kube_app set status='apply-failed' where status='applying';"

sudo -u postgres psql -d sysinv -c "update kube_app set status='remove-failed' where status='removing';"

You can then retrigger the application apply/remove.

Tee

From: Xu, Chenjie [mailto:chenjie.xu@intel.com]
Sent: June-28-19 3:53 AM
To: Lin, Shuicheng; Ezpeer Chen; 张鲲鹏
Cc: starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] [stx-config] platform-integ-apps was applying with a long time

Hi Kunpeng,

I met the same issue before. As a result of poweroff, the status will always be applying because the job has been killed and the status in the database won’t be changed from applying to apply-failed. What’s more, you can’t re-apply because the status is applying.

I changed the status in the database and then re-apply. Hope the following commands can be useful:

cat /etc/sysinv/sysinv.conf | grep sql

connection=postgresql+psycopg2://admin-sysinv:d0df960de712Ti0*@192.168.204.2/sysinv

psql -h 192.168.204.2 -U admin-sysinv -W sysinv

       d0df960de712Ti0*

select * from kube_app;

update kube_app set status=’apply-failed’ where name=’platform-integ-apps’;

select * from kube_app;

\q

system application-apply platform-integ-apps

Best Regards,

Xu, Chenjie

From: Lin, Shuicheng [mailto:shuicheng.lin@intel.com]
Sent: Friday, June 28, 2019 3:43 PM
To: Ezpeer Chen <ezpeerchen@gmail.com>; 张鲲鹏 <zhang.kunpeng@99cloud.net>
Cc: starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] [stx-config] platform-integ-apps was applying with a long time

Hi Kunpeng,

You could check /var/log/armada/platform-integ-apps-apply.log, to get details where the process stuck at.

Best Regards

Shuicheng

From: Ezpeer Chen [mailto:ezpeerchen@gmail.com]
Sent: Friday, June 28, 2019 3:30 PM
To: 张鲲鹏 <zhang.kunpeng@99cloud.net>
Cc: starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] [stx-config] platform-integ-apps was applying with a long time

Dear Kunpeng,

Your progress status is applying.

If it shows apply-failed status, then do re-apply.

Command:

# system application-apply platform-integ-apps

Thanks

张鲲鹏 <zhang.kunpeng@99cloud.net> 於 2019年6月28日 週五 上午11:48寫道：

Hi guys,

When I use sysinv to bring up the containerized services, platform-integ-apps is stay applying for a long time. I wonder it may be caused by recent poweroff. Now I have reactived the controllers, and can you tell me how to re-apply it?

[root@controller-0 log(keystone_admin)]# system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname     | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1  | controller-0 | controller  | unlocked       | enabled     | available    |
| 2  | controller-1 | controller  | unlocked       | enabled     | available    |
+----+--------------+-------------+----------------+-------------+--------------+
[root@controller-0 log(keystone_admin)]# system application-list
+---------------------+---------------------------+-------------------------------+--------------------+-----------+------------------------------------------------------------------+
| application         | version                   | manifest name                 | manifest file      | status    | progress                                                         |
+---------------------+---------------------------+-------------------------------+--------------------+-----------+------------------------------------------------------------------+
| platform-integ-apps | 1.0-7                     | platform-integration-manifest | manifest.yaml      | applying  | processing chart: stx-rbd-provisioner, overall completion: 50.0% |
| stx-openstack       | 1.0-16-centos-stable-     | armada-manifest               | stx-openstack.yaml | uploading | validating and uploading charts                                  |
|                     | latest                    |                               |                    |           |                                                                  |
|                     |                           |                               |                    |           |                                                                  |
+---------------------+---------------------------+-------------------------------+--------------------+-----------+—————————————————————————————————+

Thanks
Kunpeng
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss