Hi,

I think the likely reason you got the OrphanedObjectError exception is because you were trying to save new app state to the database without a context. Other observations:
- Checking if kubernetes is enabled is unnecessary. Kubernetes is always enabled in StarlingX.

- By default, enabled logging levels are INFO and higher

- Currently only helm chart based apps can be managed via system application commands.

- The rpc_app name is not suitable as the app object did not come from RPC call.

Tee

From: Chen, Haochuan Z [mailto:haochuan.z.chen@intel.com]
Sent: June-05-19 9:29 AM
To: Ngo, Tee; Bailey, Henry Albert (Al); starlingx-discuss@lists.starlingx.io
Subject: fix solution proposal for LP1826047

Hi Ngo & Bailey

Today I add these lines in stx-config/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py, function __init__

system = dbapi.isystem_get_one()

if system.capabilities.get('kubernetes_enabled', False):

LOG.error("kubernetes_enabled")

for rpc_app in dbapi.kube_app_get_all():

app = AppOperator.Application(rpc_app,

rpc_app.get('name') in self._helm.get_helm_applications())

op = ""

if app.status == constants.APP_APPLY_IN_PROGRESS:

op = constants.APP_APPLY_OP

elif app.status == constants.APP_UPLOAD_IN_PROGRESS:

op = constants.APP_UPLOAD_OP

elif app.status == constants.APP_REMOVE_IN_PROGRESS:

op = constants.APP_REMOVE_OP

if op:

self._abort_operation(app, op)

LOG.info("reset %s to %s failed" % (app.name, op))

But it will fail with

2019-06-05 07:54:11.549 1987918 WARNING ceph_client [-] skip checking server certificate

2019-06-05 07:54:12.055 1987918 ERROR sysinv.conductor.kube_app [-] kubernetes_enabled

2019-06-05 07:54:12.062 1987918 ERROR sysinv.openstack.common.threadgroup [-] Cannot call save on orphaned KubeApp object

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup Traceback (most recent call last):

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 117, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup x.wait()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 49, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.thread.wait()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self._exit_event.wait()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return hubs.get_hub().switch()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.greenlet.switch()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup result = function(*args, **kwargs)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/service.py", line 450, in run_service

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup service.start()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 179, in start

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._start()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 205, in _start

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._app = kube_app.AppOperator(self.dbapi)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 157, in __init__

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._abort_operation(app, op)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 198, in _abort_operation

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup progress)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 189, in _update_app_status

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup app.update_status(new_status, new_progress)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1415, in update_status

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._kube_app.save()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/objects/base.py", line 128, in wrapper

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup objtype=self.obj_name())

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup OrphanedObjectError: Cannot call save on orphaned KubeApp object

Any idea about it, or whether this solution is feasible or not.

Now I work on perform_apply_abort, tomorrow I will update to you.

And for sysinv-api exit, it should doesn’t matter. As sysinv-conductor takes the task to make application apply and status update in database, sysinv-api just polling database with application-list

BR!

Martin, Chen

SSP, Software Engineer

021-61164330

From: Ngo, Tee [mailto:Tee.Ngo@windriver.com]
Sent: Tuesday, June 4, 2019 9:27 PM
To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: fix solution proposal for LP1826047

It should be a combination of 2 solutions as solution #2 provides the option to bail while sysinv processes are running

a) application thread stalled (e.g. during the cleanup of (test) mariadb pods/stx-openstack namespace issue seen in the past and the system became sluggish)

b) client simply wants to abort the operation

Solution #2 is more involved than just updating the app state in the database.

Tee

From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com]
Sent: June-04-19 8:52 AM
To: Chen, Haochuan Z; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] fix solution proposal for LP1826047

Solution 1 sounds like how Openstack Heat handles an abnormal termination of the heat-engine (how it needs to properly clean up its stack locks)

Solution 2 is similar to what I have seen for nova and cinder CLI options.

Personally, I like option 1.

From: Chen, Haochuan Z [mailto:haochuan.z.chen@intel.com]
Sent: Tuesday, June 04, 2019 1:28 AM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] fix solution proposal for LP1826047

I am check this launch pad issue. This issue is failed, when system application apply(retrieving docker image), sysinv-api exit unexpected. When service manager re-launch sysinv-api again, application status is in applying status.

So application could not be removed. There are two solutions.

Solution 1, when sysinv-api or sysinv-conductor launch, in __init__ function, check application status in database, if status is “uploading”, ”applying” or “removing”, change the status to “upload-failed”, “apply-failed” or “removed-failed”

Solution 2, add perform-abort action for upload or apply. Use a flag to quickly exit and upload and apply action, and set database to “upload-failed” or “apply-failed”.

https://bugs.launchpad.net/starlingx/+bug/1826047

Wait for brain-storming.

Thanks!

Martin, Chen

SSP, Software Engineer

021-61164330