[Starlingx-discuss] fix solution proposal for LP1826047

Ngo, Tee Tee.Ngo at windriver.com
Wed Jun 5 16:02:34 UTC 2019 

Hi,

I think the likely reason you got the OrphanedObjectError exception is because you were trying to save new app state to the database without a context. Other observations:
  - Checking if kubernetes is enabled is unnecessary. Kubernetes is always enabled in StarlingX.
  - By default, enabled logging levels are INFO and higher
  - Currently only helm chart based apps can be managed via system application commands.
  - The rpc_app name is not suitable as the app object did not come from RPC call.

Tee

From: Chen, Haochuan Z [mailto:haochuan.z.chen at intel.com]
Sent: June-05-19 9:29 AM
To: Ngo, Tee; Bailey, Henry Albert (Al); starlingx-discuss at lists.starlingx.io
Subject: fix solution proposal for LP1826047

Hi Ngo & Bailey

Today I add these lines in stx-config/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py, function __init__
        system = dbapi.isystem_get_one()
        if system.capabilities.get('kubernetes_enabled', False):
            LOG.error("kubernetes_enabled")
            for rpc_app in dbapi.kube_app_get_all():
                app = AppOperator.Application(rpc_app,
                    rpc_app.get('name') in self._helm.get_helm_applications())
                op = ""
                if app.status == constants.APP_APPLY_IN_PROGRESS:
                     op = constants.APP_APPLY_OP
                elif app.status == constants.APP_UPLOAD_IN_PROGRESS:
                     op = constants.APP_UPLOAD_OP
                elif app.status == constants.APP_REMOVE_IN_PROGRESS:
                     op = constants.APP_REMOVE_OP

                if op:
                    self._abort_operation(app, op)
                    LOG.info("reset %s to %s failed" % (app.name, op))

But it will fail with
2019-06-05 07:54:11.549 1987918 WARNING ceph_client [-] skip checking server certificate
2019-06-05 07:54:12.055 1987918 ERROR sysinv.conductor.kube_app [-] kubernetes_enabled
2019-06-05 07:54:12.062 1987918 ERROR sysinv.openstack.common.threadgroup [-] Cannot call save on orphaned KubeApp object
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup Traceback (most recent call last):
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 117, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     x.wait()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 49, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return self.thread.wait()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return self._exit_event.wait()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return hubs.get_hub().switch()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return self.greenlet.switch()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     result = function(*args, **kwargs)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/service.py", line 450, in run_service
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     service.start()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 179, in start
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._start()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 205, in _start
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._app = kube_app.AppOperator(self.dbapi)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 157, in __init__
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._abort_operation(app, op)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 198, in _abort_operation
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     progress)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 189, in _update_app_status
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     app.update_status(new_status, new_progress)
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1415, in update_status
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._kube_app.save()
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/objects/base.py", line 128, in wrapper
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     objtype=self.obj_name())
2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup OrphanedObjectError: Cannot call save on orphaned KubeApp object

Any idea about it, or whether this solution is feasible or not.
Now I work on perform_apply_abort, tomorrow I will update to you.

And for sysinv-api exit, it should doesn't matter. As sysinv-conductor takes the task to make application apply and status update in database, sysinv-api just polling database with application-list

BR!

Martin, Chen
SSP, Software Engineer
021-61164330

From: Ngo, Tee [mailto:Tee.Ngo at windriver.com]
Sent: Tuesday, June 4, 2019 9:27 PM
To: Bailey, Henry Albert (Al) <Al.Bailey at windriver.com>; Chen, Haochuan Z <haochuan.z.chen at intel.com>; starlingx-discuss at lists.starlingx.io
Subject: RE: fix solution proposal for LP1826047

It should be a combination of 2 solutions as solution #2 provides the option to bail while sysinv processes are running

a)      application thread stalled (e.g. during the cleanup of (test) mariadb pods/stx-openstack namespace issue seen in the past and the system became sluggish)

b)      client simply wants to abort the operation

Solution #2 is more involved than just updating the app state in the database.

Tee
From: Bailey, Henry Albert (Al) [mailto:Al.Bailey at windriver.com]
Sent: June-04-19 8:52 AM
To: Chen, Haochuan Z; starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] fix solution proposal for LP1826047

Solution 1 sounds like how Openstack Heat handles an abnormal termination of the heat-engine (how it needs to properly clean up its stack locks)
Solution 2 is similar to what I have seen for nova and cinder CLI options.

Personally, I like option 1.

Al

From: Chen, Haochuan Z [mailto:haochuan.z.chen at intel.com]
Sent: Tuesday, June 04, 2019 1:28 AM
To: starlingx-discuss at lists.starlingx.io<mailto:starlingx-discuss at lists.starlingx.io>
Subject: [Starlingx-discuss] fix solution proposal for LP1826047

Hi

I am check this launch pad issue.  This issue is failed, when system application apply(retrieving docker image), sysinv-api exit unexpected. When service manager re-launch sysinv-api again, application status is in applying status.
So application could not be removed. There are two solutions.

Solution 1, when sysinv-api or sysinv-conductor launch, in __init__ function, check application status in database, if status is "uploading", "applying" or "removing", change the status to "upload-failed", "apply-failed" or "removed-failed"

Solution 2, add perform-abort action for upload or apply. Use a flag to quickly exit and upload and apply action, and set database to "upload-failed" or "apply-failed".

https://bugs.launchpad.net/starlingx/+bug/1826047

Wait for brain-storming.

Thanks!

Martin, Chen
SSP, Software Engineer
021-61164330

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20190605/39af6073/attachment-0001.html>

More information about the Starlingx-discuss mailing list