Hi Ngo & Bailey Today I add these lines in stx-config/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py, function __init__ system = dbapi.isystem_get_one() if system.capabilities.get('kubernetes_enabled', False): LOG.error("kubernetes_enabled") for rpc_app in dbapi.kube_app_get_all(): app = AppOperator.Application(rpc_app, rpc_app.get('name') in self._helm.get_helm_applications()) op = "" if app.status == constants.APP_APPLY_IN_PROGRESS: op = constants.APP_APPLY_OP elif app.status == constants.APP_UPLOAD_IN_PROGRESS: op = constants.APP_UPLOAD_OP elif app.status == constants.APP_REMOVE_IN_PROGRESS: op = constants.APP_REMOVE_OP if op: self._abort_operation(app, op) LOG.info("reset %s to %s failed" % (app.name, op)) But it will fail with 2019-06-05 07:54:11.549 1987918 WARNING ceph_client [-] skip checking server certificate 2019-06-05 07:54:12.055 1987918 ERROR sysinv.conductor.kube_app [-] kubernetes_enabled 2019-06-05 07:54:12.062 1987918 ERROR sysinv.openstack.common.threadgroup [-] Cannot call save on orphaned KubeApp object 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup Traceback (most recent call last): 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 117, in wait 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup x.wait() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 49, in wait 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.thread.wait() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self._exit_event.wait() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return hubs.get_hub().switch() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.greenlet.switch() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup result = function(*args, **kwargs) 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/service.py", line 450, in run_service 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup service.start() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 179, in start 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._start() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 205, in _start 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._app = kube_app.AppOperator(self.dbapi) 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 157, in __init__ 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._abort_operation(app, op) 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 198, in _abort_operation 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup progress) 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 189, in _update_app_status 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup app.update_status(new_status, new_progress) 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1415, in update_status 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._kube_app.save() 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/objects/base.py", line 128, in wrapper 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup objtype=self.obj_name()) 2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup OrphanedObjectError: Cannot call save on orphaned KubeApp object Any idea about it, or whether this solution is feasible or not. Now I work on perform_apply_abort, tomorrow I will update to you. And for sysinv-api exit, it should doesn't matter. As sysinv-conductor takes the task to make application apply and status update in database, sysinv-api just polling database with application-list BR! Martin, Chen SSP, Software Engineer 021-61164330 From: Ngo, Tee [mailto:Tee.Ngo@windriver.com] Sent: Tuesday, June 4, 2019 9:27 PM To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com>; starlingx-discuss@lists.starlingx.io Subject: RE: fix solution proposal for LP1826047 It should be a combination of 2 solutions as solution #2 provides the option to bail while sysinv processes are running a) application thread stalled (e.g. during the cleanup of (test) mariadb pods/stx-openstack namespace issue seen in the past and the system became sluggish) b) client simply wants to abort the operation Solution #2 is more involved than just updating the app state in the database. Tee From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com] Sent: June-04-19 8:52 AM To: Chen, Haochuan Z; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: Re: [Starlingx-discuss] fix solution proposal for LP1826047 Solution 1 sounds like how Openstack Heat handles an abnormal termination of the heat-engine (how it needs to properly clean up its stack locks) Solution 2 is similar to what I have seen for nova and cinder CLI options. Personally, I like option 1. Al From: Chen, Haochuan Z [mailto:haochuan.z.chen@intel.com] Sent: Tuesday, June 04, 2019 1:28 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> Subject: [Starlingx-discuss] fix solution proposal for LP1826047 Hi I am check this launch pad issue. This issue is failed, when system application apply(retrieving docker image), sysinv-api exit unexpected. When service manager re-launch sysinv-api again, application status is in applying status. So application could not be removed. There are two solutions. Solution 1, when sysinv-api or sysinv-conductor launch, in __init__ function, check application status in database, if status is "uploading", "applying" or "removing", change the status to "upload-failed", "apply-failed" or "removed-failed" Solution 2, add perform-abort action for upload or apply. Use a flag to quickly exit and upload and apply action, and set database to "upload-failed" or "apply-failed". https://bugs.launchpad.net/starlingx/+bug/1826047 Wait for brain-storming. Thanks! Martin, Chen SSP, Software Engineer 021-61164330