Hi Ngo & Bailey

 

Today I add these lines in stx-config/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py, function __init__

        system = dbapi.isystem_get_one()

        if system.capabilities.get('kubernetes_enabled', False):

            LOG.error("kubernetes_enabled")

            for rpc_app in dbapi.kube_app_get_all():

                app = AppOperator.Application(rpc_app,

                    rpc_app.get('name') in self._helm.get_helm_applications())

                op = ""

                if app.status == constants.APP_APPLY_IN_PROGRESS:

                     op = constants.APP_APPLY_OP

                elif app.status == constants.APP_UPLOAD_IN_PROGRESS:

                     op = constants.APP_UPLOAD_OP

                elif app.status == constants.APP_REMOVE_IN_PROGRESS:

                     op = constants.APP_REMOVE_OP

 

                if op:

                    self._abort_operation(app, op)

                    LOG.info("reset %s to %s failed" % (app.name, op))

 

But it will fail with

2019-06-05 07:54:11.549 1987918 WARNING ceph_client [-] skip checking server certificate

2019-06-05 07:54:12.055 1987918 ERROR sysinv.conductor.kube_app [-] kubernetes_enabled

2019-06-05 07:54:12.062 1987918 ERROR sysinv.openstack.common.threadgroup [-] Cannot call save on orphaned KubeApp object

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup Traceback (most recent call last):

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 117, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     x.wait()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 49, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return self.thread.wait()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return self._exit_event.wait()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return hubs.get_hub().switch()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     return self.greenlet.switch()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     result = function(*args, **kwargs)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/service.py", line 450, in run_service

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     service.start()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 179, in start

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._start()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 205, in _start

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._app = kube_app.AppOperator(self.dbapi)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 157, in __init__

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._abort_operation(app, op)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 198, in _abort_operation

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     progress)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 189, in _update_app_status

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     app.update_status(new_status, new_progress)

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1415, in update_status

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     self._kube_app.save()

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/sysinv/objects/base.py", line 128, in wrapper

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup     objtype=self.obj_name())

2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup OrphanedObjectError: Cannot call save on orphaned KubeApp object

 

Any idea about it, or whether this solution is feasible or not.

Now I work on perform_apply_abort, tomorrow I will update to you.

 

And for sysinv-api exit, it should doesn’t matter. As sysinv-conductor takes the task to make application apply and status update in database, sysinv-api just polling database with application-list

 

BR!

 

Martin, Chen

SSP, Software Engineer

021-61164330

 

From: Ngo, Tee [mailto:Tee.Ngo@windriver.com]
Sent: Tuesday, June 4, 2019 9:27 PM
To: Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com>; starlingx-discuss@lists.starlingx.io
Subject: RE: fix solution proposal for LP1826047

 

It should be a combination of 2 solutions as solution #2 provides the option to bail while sysinv processes are running

a)      application thread stalled (e.g. during the cleanup of (test) mariadb pods/stx-openstack namespace issue seen in the past and the system became sluggish)

b)      client simply wants to abort the operation

 

Solution #2 is more involved than just updating the app state in the database.

 

Tee

From: Bailey, Henry Albert (Al) [mailto:Al.Bailey@windriver.com]
Sent: June-04-19 8:52 AM
To: Chen, Haochuan Z; starlingx-discuss@lists.starlingx.io
Subject: Re: [Starlingx-discuss] fix solution proposal for LP1826047

 

Solution 1 sounds like how Openstack Heat handles an abnormal termination of the heat-engine (how it needs to properly clean up its stack locks)

Solution 2 is similar to what I have seen for nova and cinder CLI options.

 

Personally, I like option 1.

 

Al

 

From: Chen, Haochuan Z [mailto:haochuan.z.chen@intel.com]
Sent: Tuesday, June 04, 2019 1:28 AM
To: starlingx-discuss@lists.starlingx.io
Subject: [Starlingx-discuss] fix solution proposal for LP1826047

 

Hi

 

I am check this launch pad issue.  This issue is failed, when system application apply(retrieving docker image), sysinv-api exit unexpected. When service manager re-launch sysinv-api again, application status is in applying status.

So application could not be removed. There are two solutions.

 

Solution 1, when sysinv-api or sysinv-conductor launch, in __init__ function, check application status in database, if status is “uploading”, ”applying” or “removing”, change the status to “upload-failed”, “apply-failed” or “removed-failed”

 

Solution 2, add perform-abort action for upload or apply. Use a flag to quickly exit and upload and apply action, and set database to “upload-failed” or “apply-failed”.

 

https://bugs.launchpad.net/starlingx/+bug/1826047

 

Wait for brain-storming.

 

Thanks!

 

Martin, Chen

SSP, Software Engineer

021-61164330