Today I add these lines in stx-config/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py, function __init__
system = dbapi.isystem_get_one()
if system.capabilities.get('kubernetes_enabled', False):
LOG.error("kubernetes_enabled")
for rpc_app in dbapi.kube_app_get_all():
app = AppOperator.Application(rpc_app,
rpc_app.get('name') in self._helm.get_helm_applications())
op = ""
if app.status == constants.APP_APPLY_IN_PROGRESS:
op = constants.APP_APPLY_OP
elif app.status == constants.APP_UPLOAD_IN_PROGRESS:
op = constants.APP_UPLOAD_OP
elif app.status == constants.APP_REMOVE_IN_PROGRESS:
op = constants.APP_REMOVE_OP
if op:
self._abort_operation(app, op)
LOG.info("reset %s to %s failed" % (app.name, op))
But it will fail with
2019-06-05 07:54:11.549 1987918 WARNING ceph_client [-] skip checking server certificate
2019-06-05 07:54:12.055 1987918 ERROR sysinv.conductor.kube_app [-] kubernetes_enabled
2019-06-05 07:54:12.062 1987918 ERROR sysinv.openstack.common.threadgroup [-] Cannot call save on orphaned KubeApp object
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup Traceback (most recent call last):
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 117, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup x.wait()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 49, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.thread.wait()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self._exit_event.wait()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return hubs.get_hub().switch()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.greenlet.switch()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup result = function(*args, **kwargs)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/service.py", line 450, in run_service
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup service.start()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 179, in start
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._start()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 205, in _start
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._app = kube_app.AppOperator(self.dbapi)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 157, in __init__
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._abort_operation(app, op)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 198, in _abort_operation
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup progress)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 189, in _update_app_status
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup app.update_status(new_status, new_progress)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1415, in update_status
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._kube_app.save()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/objects/base.py", line 128, in wrapper
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup objtype=self.obj_name())
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup
OrphanedObjectError: Cannot call save on orphaned KubeApp object
Any idea about it, or whether this solution is feasible or not.
Now I work on perform_apply_abort, tomorrow I will update to you.
And for sysinv-api exit, it should doesn’t matter. As sysinv-conductor takes the task to make application apply and status update in database, sysinv-api just polling database with application-list
BR!
Martin, Chen
SSP, Software Engineer
021-61164330
From: Ngo, Tee [mailto:Tee.Ngo@windriver.com]
Sent: Tuesday, June 4, 2019 9:27 PM
To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: fix solution proposal for LP1826047
It should be a combination of 2 solutions as solution #2 provides the option to bail while sysinv processes are running
a)
application thread stalled (e.g. during the cleanup of (test) mariadb pods/stx-openstack namespace issue seen in the past and the system became sluggish)
b)
client simply wants to abort the operation
Solution #2 is more involved than just updating the app state in the database.
Tee
From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com]
Sent: June-04-19 8:52 AM
To: Chen, Haochuan Z;
starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] fix solution proposal for LP1826047
Solution 1 sounds like how Openstack Heat handles an abnormal termination of the heat-engine (how it needs to properly clean up its stack locks)
Solution 2 is similar to what I have seen for nova and cinder CLI options.
Personally, I like option 1.
Al
From: Chen, Haochuan Z [mailto:haochuan.z.chen@intel.com]
Sent: Tuesday, June 04, 2019 1:28 AM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] fix solution proposal for LP1826047
Hi
I am check this launch pad issue. This issue is failed, when system application apply(retrieving docker image), sysinv-api exit unexpected. When service manager re-launch sysinv-api again, application status is in applying status.
So application could not be removed. There are two solutions.
Solution 1, when sysinv-api or sysinv-conductor launch, in __init__ function, check application status in database, if status is “uploading”, ”applying” or “removing”, change the status to “upload-failed”, “apply-failed” or “removed-failed”
Solution 2, add perform-abort action for upload or apply. Use a flag to quickly exit and upload and apply action, and set database to “upload-failed” or “apply-failed”.
https://bugs.launchpad.net/starlingx/+bug/1826047
Wait for brain-storming.
Thanks!
Martin, Chen
SSP, Software Engineer
021-61164330