<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello Martin,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
No, it is the first time seeing it.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
But I see that logic is introduced by<br>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Project: starlingx/ansible-playbooks<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><span>Commit 514d4e7262f80a73ab37e0132f9e3b30088d14ad<br>
</span><span></span></span><span>CommitDate: Wed Jun 10 13:17:00 2020 -0400<br>
</span><span></span><br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText">Thanks,<br>
Dan Voiculeasa</div>
</span></font></div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Chen, Haochuan Z <haochuan.z.chen@intel.com><br>
<b>Sent:</b> Thursday, June 18, 2020 4:18 AM<br>
<b>To:</b> Voiculeasa, Dan <Dan.Voiculeasa@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io><br>
<b>Subject:</b> RE: issue for backup and restore</font>
<div> </div>
</div>
<style>
<!--
@font-face
        {font-family:SimSun}
@font-face
        {font-family:"Cambria Math"}
@font-face
        {font-family:DengXian}
@font-face
        {font-family:Calibri}
@font-face
        {}
@font-face
        {}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
        {color:#0563C1;
        text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
        {color:#954F72;
        text-decoration:underline}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xmsonormal0, li.x_xmsonormal0, div.x_xmsonormal0
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xxmsonormal, li.x_xxmsonormal, div.x_xxmsonormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xxmsochpdefault, li.x_xxmsochpdefault, div.x_xxmsochpdefault
        {margin-right:0in;
        margin-left:0in;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif}
p.x_xmsochpdefault, li.x_xmsochpdefault, div.x_xmsochpdefault
        {margin-right:0in;
        margin-left:0in;
        font-size:10.0pt;
        font-family:"Calibri",sans-serif}
span.x_xmsohyperlink
        {color:#0563C1;
        text-decoration:underline}
span.x_xmsohyperlinkfollowed
        {color:#954F72;
        text-decoration:underline}
span.x_xxmsohyperlink
        {color:#0563C1;
        text-decoration:underline}
span.x_xxmsohyperlinkfollowed
        {color:#954F72;
        text-decoration:underline}
span.x_xxemailstyle17
        {font-family:"Calibri",sans-serif;
        color:windowtext}
span.x_xemailstyle25
        {font-family:"Calibri",sans-serif;
        color:windowtext}
span.x_EmailStyle29
        {font-family:"Calibri",sans-serif;
        color:windowtext}
span.x_EmailStyle31
        {font-family:"Calibri",sans-serif;
        color:windowtext}
.x_MsoChpDefault
        {font-size:10.0pt}
@page WordSection1
        {margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
        {}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="x_WordSection1">
<p class="x_MsoNormal">Hi Dan</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I check restore for latest code, restore will fail with such log. I used to check code base Jun 5 master branch, no such issue.</p>
<p class="x_MsoNormal">You know about this?</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">TASK [bootstrap/bringup-essential-services : Create Armada node label] **********************************************************************************************************************</p>
<p class="x_MsoNormal">fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["kubectl", "label", "node", "controller-0", "armada=enabled"], "delta": "0:00:00.102152", "end": "2020-06-18 00:57:32.563552", "msg": "non-zero return code", "rc": 1, "start": "2020-06-18
 00:57:32.461400", "stderr": "error: 'armada' already has a value (enabled), and --overwrite is false", "stderr_lines": ["error: 'armada' already has a value (enabled), and --overwrite is false"], "stdout": "", "stdout_lines": []}</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">PLAY RECAP **********************************************************************************************************************************************************************************</p>
<p class="x_MsoNormal">localhost                  : ok=354  changed=156  unreachable=0    failed=1</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">[sysadmin@controller-0 ~(keystone_admin)]$</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">BR!</p>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal">Martin, Chen</p>
<p class="x_MsoNormal">IOTG, Software Engineer</p>
<p class="x_MsoNormal">021-61164330</p>
</div>
<p class="x_MsoNormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b>From:</b> Chen, Haochuan Z <br>
<b>Sent:</b> Thursday, June 11, 2020 10:53 AM<br>
<b>To:</b> Voiculeasa, Dan <Dan.Voiculeasa@windriver.com>; starlingx-discuss@lists.starlingx.io<br>
<b>Subject:</b> RE: issue for backup and restore</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Hi voiculeasa</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I confirm backup and restore works without ceph backend.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">This issue is caused with my improper provision step.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">BR!</p>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal">Martin, Chen</p>
<p class="x_MsoNormal">IOTG, Software Engineer</p>
<p class="x_MsoNormal">021-61164330</p>
</div>
<p class="x_MsoNormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b>From:</b> Voiculeasa, Dan <<a href="mailto:Dan.Voiculeasa@windriver.com">Dan.Voiculeasa@windriver.com</a>>
<br>
<b>Sent:</b> Tuesday, June 9, 2020 5:54 PM<br>
<b>To:</b> Chen, Haochuan Z <<a href="mailto:haochuan.z.chen@intel.com">haochuan.z.chen@intel.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> Re: issue for backup and restore</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Hello Martin,</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">I didn't encounter that issue when testing, but also I didn't test recently without ceph backend.</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Are you using a local build iso? Are you testing some change in the source code? Any prior successful restore on a simplex with ceph / simplex without ceph?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div id="x_Signature">
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:10.0pt">Thanks,<br>
Dan Voiculeasa</span></p>
</div>
</div>
</div>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Chen, Haochuan Z <<a href="mailto:haochuan.z.chen@intel.com">haochuan.z.chen@intel.com</a>><br>
<b>Sent:</b> Monday, June 8, 2020 5:21 AM<br>
<b>To:</b> Voiculeasa, Dan <<a href="mailto:Dan.Voiculeasa@windriver.com">Dan.Voiculeasa@windriver.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a> <<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Subject:</b> Re: [Starlingx-discuss] issue for backup and restore</span> </p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal">Hi voiculeasa</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">When you restore system, do you have such issue. I deploy the system without add storagebackend ceph, simplex.</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Restore process</p>
<p class="x_xmsonormal">$ sudo ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "initial_backup_dir=/home/sysadmin ansible_become_pass=Local.123 admin_password=Local.123 backup_filename=localhost_platform_backup_2020_06_08_00_25_30.tgz"</p>
<p class="x_xmsonormal">$ source /etc/platform/openrc</p>
<p class="x_xmsonormal">$ system host-unlock 1</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">u'9\nTraceback (most recent call last):\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 437, in _process_data\n    **args)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py",
 line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1691, in configure_ihost\n    self._configure_controller_host(context, host)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py",
 line 1325, in _configure_controller_host\n    self._puppet.update_host_config(host)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py", line 31, in _wrapper\n    func(self, *args, **kwargs)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/puppet.py",
 line 148, in update_host_config\n    config.update(puppet_plugin.obj.get_host_config(host))\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 111, in get_host_config\n    generate_driver_config(context, config)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py",
 line 1412, in generate_driver_config\n    generate_mlx4_core_options(context, config)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 1389, in generate_mlx4_core_options\n    num_vfs_options = build_mlx4_num_vfs_options(context)\n\n 
 File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 1358, in build_mlx4_num_vfs_options\n    ifaces = find_sriov_interfaces_by_driver(context, constants.DRIVER_MLX_CX3)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py",
 line 1264, in find_sriov_interfaces_by_driver\n    port = get_interface_port(context, iface)\n\n  File "/usr/lib64/python2.7/site-packages/sysinv/puppet/interface.py", line 515, in get_interface_port\n    return interface.get_interface_port(context, iface)\n\n 
 File "/usr/lib64/python2.7/site-packages/sysinv/common/interface.py", line 105, in get_interface_port\n    return context[\'ports\'][iface[\'id\']]\n\nKeyError: 9\n'</p>
<p class="x_xmsonormal">[sysadmin@localhost playbooks(keystone_admin)]$</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">Thanks</p>
<p class="x_xmsonormal"> </p>
<div>
<p class="x_xmsonormal">Martin, Chen</p>
<p class="x_xmsonormal">IOTG, Software Engineer</p>
<p class="x_xmsonormal">021-61164330</p>
</div>
<p class="x_xmsonormal"> </p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_xmsonormal"><b>From:</b> Voiculeasa, Dan <<a href="mailto:Dan.Voiculeasa@windriver.com">Dan.Voiculeasa@windriver.com</a>>
<br>
<b>Sent:</b> Tuesday, June 2, 2020 9:23 PM<br>
<b>To:</b> Chen, Haochuan Z <<a href="mailto:haochuan.z.chen@intel.com">haochuan.z.chen@intel.com</a>>;
<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a><br>
<b>Subject:</b> Re: issue for backup and restore</p>
</div>
</div>
<p class="x_xmsonormal"> </p>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">Hello,</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">What does /var/log/puppet/2020-05-24-12-50-51_controller/puppet.log say?</span></p>
</div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt; color:black">If you have the setup in the reproduced state, send me a zoom meeting invite starting in the interval [now, +7 hours].</span></p>
</div>
</div>
</div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:12.0pt"> </span></p>
</div>
<div>
<div>
<div>
<p class="x_xmsonormal"><span style="font-size:10.0pt">Thanks,<br>
Dan Voiculeasa</span></p>
</div>
</div>
</div>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="x_x_divRplyFwdMsg">
<p class="x_xmsonormal"><b><span style="color:black">From:</span></b><span style="color:black"> Chen, Haochuan Z <<a href="mailto:haochuan.z.chen@intel.com">haochuan.z.chen@intel.com</a>><br>
<b>Sent:</b> Sunday, May 24, 2020 4:08 PM<br>
<b>To:</b> <a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a> <<a href="mailto:starlingx-discuss@lists.starlingx.io">starlingx-discuss@lists.starlingx.io</a>><br>
<b>Subject:</b> [Starlingx-discuss] issue for backup and restore</span> </p>
<div>
<p class="x_xmsonormal"> </p>
</div>
</div>
<div>
<div>
<p class="x_xxmsonormal">Hi</p>
<p class="x_xxmsonormal"> </p>
<p class="x_xxmsonormal">I follow this guide to check backup and restore</p>
<p class="x_xxmsonormal"><a href="https://opendev.org/starlingx/docs/src/commit/adc24ba565a58cf7e20639d4630d6d1893337bbb/doc/source/developer_resources/backup_restore.rst">https://opendev.org/starlingx/docs/src/commit/adc24ba565a58cf7e20639d4630d6d1893337bbb/doc/source/developer_resources/backup_restore.rst</a></p>
<p class="x_xxmsonormal"> </p>
<p class="x_xxmsonormal">But when I run this command to restore the system, it will fail with such error log.</p>
<p class="x_xxmsonormal">sudo ansible-playbook /usr/share/ansible/stx-ansible/playbooks/restore_platform.yml -e "initial_backup_dir=/home/sysadmin ansible_become_pass=sysadmin admin_password=sysadmin backup_filename=localhost_platform_backup_2020_05_23_23_43_40.tgz"</p>
<p class="x_xxmsonormal"> </p>
<p class="x_xxmsonormal">TASK [bootstrap/apply-bootstrap-manifest : Applying puppet bootstrap manifest] *******************************************************************************</p>
<p class="x_xxmsonormal">fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["/usr/local/bin/puppet-manifest-apply.sh", "/tmp/hieradata", "192.188.204.3", "controller", "ansible_bootstrap", ">", "/tmp/apply_manifest.log"], "delta": "0:02:18.526028", "end":
 "2020-05-24 12:53:09.811312", "msg": "non-zero return code", "rc": 1, "start": "2020-05-24 12:50:51.285284", "stderr": "cp: cannot stat ‘/tmp/hieradata/192.188.204.3.yaml’: No such file or directory\ncp: cannot stat ‘/tmp/hieradata/system.yaml’: No such file
 or directory\ncp: cannot stat ‘/tmp/hieradata/secure_system.yaml’: No such file or directory\ncp: cannot stat ‘>’: No such file or directory", "stderr_lines": ["cp: cannot stat ‘/tmp/hieradata/192.188.204.3.yaml’: No such file or directory", "cp: cannot stat
 ‘/tmp/hieradata/system.yaml’: No such file or directory", "cp: cannot stat ‘/tmp/hieradata/secure_system.yaml’: No such file or directory", "cp: cannot stat ‘>’: No such file or directory"], "stdout": "Applying puppet ansible_bootstrap manifest...\n[WARNING]\nWarnings
 found. See /var/log/puppet/2020-05-24-12-50-51_controller/puppet.log for details", "stdout_lines": ["Applying puppet ansible_bootstrap manifest...", "[WARNING]", "Warnings found. See /var/log/puppet/2020-05-24-12-50-51_controller/puppet.log for details"]}</p>
<p class="x_xxmsonormal"> </p>
<p class="x_xxmsonormal">Any idea about this.</p>
<p class="x_xxmsonormal"> </p>
<p class="x_xxmsonormal">Thanks!</p>
<p class="x_xxmsonormal"> </p>
<p class="x_xxmsonormal">Martin, Chen</p>
<p class="x_xxmsonormal">IOTG, Software Engineer</p>
<p class="x_xxmsonormal">021-61164330</p>
<p class="x_xxmsonormal"> </p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>