<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:#0563C1;
        text-decoration:underline;}
span.EmailStyle17
        {mso-style-type:personal-compose;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
span.error
        {mso-style-name:error;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-family:"Calibri",sans-serif;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi we are seeing the node switchover failure<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoTableGrid" border="1" cellspacing="0" cellpadding="0" style="border-collapse:collapse;border:none">
<tbody>
<tr>
<td width="1848" valign="top" style="width:1385.7pt;border:solid windowtext 1.0pt;background:#D0CECE;padding:0in 5.4pt 0in 5.4pt">
<p><strong><span style="font-family:"Calibri",sans-serif">Issue:</span></strong><span style="color:black"> Switch active controller action in Central cloud is failing. It is not showing any failure in StarlingX GUI but after progressing for a while, it returns
 to the old state. i.e. same controller node remains in Active state. </span><o:p></o:p></p>
<p><strong><span style="font-family:"Calibri",sans-serif;color:black">Steps to reproduce:</span></strong><o:p></o:p></p>
<p><span style="color:black">(1) Under Hosts tab for Active controller, click on Actions dropdown and select "swact host"<br>
(2) Verify Controller-0 and Controller-1 Personality in the Host Inventory section</span><o:p></o:p></p>
<p><strong><span style="font-family:"Calibri",sans-serif;color:black">Expected Result:</span></strong><o:p></o:p></p>
<p><span style="color:black">(1) At step 5, Swact host action should be successful. <br>
(2) At step 6, whichever controller was active before should be displayed as Standby now</span><o:p></o:p></p>
<p><strong><span style="font-family:"Calibri",sans-serif;color:black">Note:</span></strong><span style="color:black"> StarlingX GUI issue screenshots are attached</span><o:p></o:p></p>
<p><span style="color:black">From sysinv.log,</span><o:p></o:p></p>
<p><span style="color:black">sysinv 2021-03-29 12:35:48.468 29411 INFO sysinv.api.controllers.v1.host
<span class="error">[-]</span> controller-1 1. delta_handle <span class="error">['action']</span><br>
sysinv 2021-03-29 12:35:48.850 29411 INFO sysinv.api.controllers.v1.rest_api <span class="error">
[-]</span> PATCH cmd:<a href="http://controller-1:7777/v1/servicenode/controller-1" target="_blank">http://controller-1:7777/v1/servicenode/controller-1</a> hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"origin": "sysinv", "action":
 "swact-pre-check", "admin": "unknown", "oper": "unknown", "avail": ""}<br>
sysinv 2021-03-29 12:35:48.940 29411 INFO sysinv.api.controllers.v1.rest_api <span class="error">
[-]</span> Response={u'origin': u'sm', u'oper': u'unknown', u'admin': u'unknown', u'hostname': u'controller-1', u'avail': u'', u'error_details': None, u'action': u'swact-pre-check', u'error_code': u'0'}<br>
sysinv 2021-03-29 12:35:48.942 29411 INFO sysinv.api.controllers.v1.host <span class="error">
[-]</span> controller-1 Action staged: swact<br>
sysinv 2021-03-29 12:35:48.942 29411 INFO sysinv.api.controllers.v1.host <span class="error">
[-]</span> controller-1 post action_stage hostupdate action=swact notify_vim=False notify_mtc=True skip_notify_mtce=False<br>
sysinv 2021-03-29 12:35:48.942 29411 INFO sysinv.api.controllers.v1.host <span class="error">
[-]</span> controller-1 2. delta_handle <span class="error">['action']</span><br>
sysinv 2021-03-29 12:35:48.943 29411 INFO sysinv.api.controllers.v1.host <span class="error">
[-]</span> controller-1 apply ihost_val {'task': u'Swacting'}<br>
sysinv 2021-03-29 12:35:48.957 29411 INFO sysinv.api.controllers.v1.host <span class="error">
[-]</span> controller-1 Action swact perform notify_mtce<br>
cmd:<a href="http://localhost:2112/v1/hosts/6cf35736-20dd-4921-a5ed-faebbfa036b4" target="_blank">http://localhost:2112/v1/hosts/6cf35736-20dd-4921-a5ed-faebbfa036b4</a> hdr:{'Content-type': 'application/json', 'User-Agent': 'sysinv/1.0'} payload:{"tboot":
 "false", "ttys_dcd": null, "subfunctions": "controller,worker,lowlatency", "bm_ip": null, "install_state": "completed+", "rootfs_device": "/dev/sda", "bm_username": null, "clock_synchronization": "ntp", "operation": "modify", "serialid": null, "id": 2, "console":
 "ttyS0,115200", "uuid": "6cf35736-20dd-4921-a5ed-faebbfa036b4", "mgmt_ip": "10.222.21.3", "software_load": "20.06", "config_status": null, "hostname": "controller-1", "iscsi_initiator_name": "iqn.1994-05.com.redhat:4e6911b5176d", "capabilities":</span><o:p></o:p></p>
<p><span style="color:black">{"stor_function": "monitor"}</span><o:p></o:p></p>
<p><span style="color:black">, "install_output": "text", "device_image_update": null, "location": {}, "availability": "available", "invprovision": "provisioned", "peer_id": null, "administrative": "unlocked", "personality": "controller", "recordtype": "standard",
 "reboot_needed": false, "bm_mac": null, "inv_state": "inventoried", "mtce_info": null, "isystem_uuid": "73b38f1a-8b20-436e-9eba-9a619a448cf4", "boot_device": "/dev/sda", "install_state_info": null, "mgmt_mac": "2c:ea:7f:65:a8:a6", "subfunction_oper": "enabled",
 "target_load": "20.06", "vsc_controllers": null, "operational": "enabled", "subfunction_avail": "available", "action": "swact", "bm_type": null}<br>
sysinv 2021-03-29 12:36:34.304 10553 ERROR sysinv.openstack.common.rpc.common <span class="error">
[-]</span> Failed to consume message from queue: <span class="error">[Errno 104]</span> Connection reset by peer: error:
<span class="error">[Errno 104]</span> Connection reset by peer</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There are some alarms raised during switchover process and it seems the peer node is not responding.
<o:p></o:p></p>
<p class="MsoNormal">It is observed that after some time the original role is restored.<o:p></o:p></p>
<p class="MsoNormal"><img border="0" width="1462" height="686" style="width:15.225in;height:7.15in" id="Picture_x0020_1" src="cid:image003.jpg@01D751A3.6A7BB410"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please let me know how to further debug the issue.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Ankush<o:p></o:p></p>
</div>
</body>
</html>