<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1245722427;
mso-list-type:hybrid;
mso-list-template-ids:-502349504 67698711 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-number-format:alpha-lower;
mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:#1F497D">Hi Ngo & Bailey<o:p></o:p></span></a></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Today I add these lines in stx-config/sysinv/sysinv/sysinv/sysinv/conductor/kube_app.py, function __init__<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> system = dbapi.isystem_get_one()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> if system.capabilities.get('kubernetes_enabled', False):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> LOG.error("kubernetes_enabled")<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> for rpc_app in dbapi.kube_app_get_all():<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> app = AppOperator.Application(rpc_app,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> rpc_app.get('name') in self._helm.get_helm_applications())<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> op = ""<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> if app.status == constants.APP_APPLY_IN_PROGRESS:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> op = constants.APP_APPLY_OP<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> elif app.status == constants.APP_UPLOAD_IN_PROGRESS:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> op = constants.APP_UPLOAD_OP<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> elif app.status == constants.APP_REMOVE_IN_PROGRESS:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> op = constants.APP_REMOVE_OP<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> if op:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> self._abort_operation(app, op)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> LOG.info("reset %s to %s failed" % (app.name, op))<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">But it will fail with<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:11.549 1987918 WARNING ceph_client [-] skip checking server certificate<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.055 1987918 ERROR sysinv.conductor.kube_app [-] kubernetes_enabled<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 ERROR sysinv.openstack.common.threadgroup [-] Cannot call save on orphaned KubeApp object<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup Traceback (most recent call last):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 117, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup x.wait()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/threadgroup.py", line 49, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.thread.wait()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self._exit_event.wait()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return hubs.get_hub().switch()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup return self.greenlet.switch()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup result = function(*args, **kwargs)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/service.py", line 450, in run_service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup service.start()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 179, in start<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._start()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 205, in _start<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._app = kube_app.AppOperator(self.dbapi)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 157, in __init__<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._abort_operation(app, op)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 198, in _abort_operation<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup progress)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 189, in _update_app_status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup app.update_status(new_status, new_progress)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1415, in update_status<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup self._kube_app.save()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/sysinv/objects/base.py", line 128, in wrapper<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup objtype=self.obj_name())<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2019-06-05 07:54:12.062 1987918 TRACE sysinv.openstack.common.threadgroup
</span><span style="color:red">OrphanedObjectError: Cannot call save on orphaned KubeApp object</span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Any idea about it, or whether this solution is feasible or not.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Now I work on perform_apply_abort, tomorrow I will update to you.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">And for sysinv-api exit, it should doesn’t matter. As sysinv-conductor takes the task to make application apply and status update in database, sysinv-api just polling database with application-list<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">BR!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:#1F497D">Martin, Chen<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">SSP, Software Engineer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">021-61164330<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Ngo, Tee [mailto:Tee.Ngo@windriver.com] <br>
<b>Sent:</b> Tuesday, June 4, 2019 9:27 PM<br>
<b>To:</b> Bailey, Henry Albert (Al) <Al.Bailey@windriver.com>; Chen, Haochuan Z <haochuan.z.chen@intel.com>; starlingx-discuss@lists.starlingx.io<br>
<b>Subject:</b> RE: fix solution proposal for LP1826047<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">It should be a combination of 2 solutions as solution #2 provides the option to bail while sysinv processes are running<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">a)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">application thread stalled (e.g. during the cleanup of (test) mariadb pods/stx-openstack namespace issue seen in the past and the system became sluggish)<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">b)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">client simply wants to abort the operation<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Solution #2 is more involved than just updating the app state in the database.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Tee<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> Bailey, Henry Albert (Al) [<a href="mailto:Al.Bailey@windriver.com">mailto:Al.Bailey@windriver.com</a>]
<br>
<b>Sent:</b> June-04-19 8:52 AM<br>
<b>To:</b> Chen, Haochuan Z; <a href="mailto:starlingx-discuss@lists.starlingx.io">
starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> Re: [Starlingx-discuss] fix solution proposal for LP1826047<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Solution 1 sounds like how Openstack Heat handles an abnormal termination of the heat-engine (how it needs to properly clean up its stack locks)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Solution 2 is similar to what I have seen for nova and cinder CLI options.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Personally, I like option 1.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Al<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> Chen, Haochuan Z [<a href="mailto:haochuan.z.chen@intel.com">mailto:haochuan.z.chen@intel.com</a>]
<br>
<b>Sent:</b> Tuesday, June 04, 2019 1:28 AM<br>
<b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> [Starlingx-discuss] fix solution proposal for LP1826047<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am check this launch pad issue. This issue is failed, when system application apply(retrieving docker image), sysinv-api exit unexpected. When service manager re-launch sysinv-api again, application status is in applying status.<o:p></o:p></p>
<p class="MsoNormal">So application could not be removed. There are two solutions.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Solution 1, when sysinv-api or sysinv-conductor launch, in __init__ function, check application status in database, if status is “uploading”, ”applying” or “removing”, change the status to “upload-failed”, “apply-failed” or “removed-failed”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Solution 2, add perform-abort action for upload or apply. Use a flag to quickly exit and upload and apply action, and set database to “upload-failed” or “apply-failed”.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a href="https://bugs.launchpad.net/starlingx/+bug/1826047">https://bugs.launchpad.net/starlingx/+bug/1826047</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Wait for brain-storming.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Martin, Chen<o:p></o:p></p>
<p class="MsoNormal">SSP, Software Engineer<o:p></o:p></p>
<p class="MsoNormal">021-61164330<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>