<div dir="ltr"><div dir="ltr">Hi Thiago,<div><br></div><div>Please find my inline comments.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 18, 2021 at 8:28 PM Brito, Thiago <<a href="mailto:Thiago.Brito@windriver.com">Thiago.Brito@windriver.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Danishka,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Looking at your logs it seems like the missing table <span style="font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)">'mysql.servers' is preventing the system to come up. Would you please raise a LP with those logs so
we can dig deeper into this problem? It would be useful to have the full logs from these attempts.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)"><br></span></div></div></blockquote><div><br></div><div>Sure, I will raise a LP.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><span style="font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)">
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)">Once you do that, you might wanna try to "application-remove" stx-openstack and check if the PVC of mariadb is still on the system with "kubectl get pvc
-n openstack". If that is the case, the app is probably trying to reuse the old "broken" database. Deleting that pvc and appling stx-openstack again should get your system up.</span></div></div></blockquote><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pvc -n openstack</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">No resources found in openstack namespace.</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n openstack -o wide</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">No resources found in openstack namespace.</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p></div><div><a href="https://paste.opendev.org/show/808178/">https://paste.opendev.org/show/808178/</a><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)">Thiago</span></div>
<div id="gmail-m_2484239224959275118appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_2484239224959275118divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>><br>
<b>Sent:</b> Wednesday, August 18, 2021 11:13 AM<br>
<b>To:</b> Pereira, Douglas <<a href="mailto:Douglas.Pereira@windriver.com" target="_blank">Douglas.Pereira@windriver.com</a>>; <a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a> <<a href="mailto:starlingx-discuss@lists.starlingx.io" target="_blank">starlingx-discuss@lists.starlingx.io</a>>; Sun, Austin <<a href="mailto:austin.sun@intel.com" target="_blank">austin.sun@intel.com</a>><br>
<b>Subject:</b> Re: [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing</font>
<div> </div>
</div>
<div>
<p></p>
<div style="border:1pt solid rgb(156,101,0);padding:2pt">
<p style="line-height:12pt;background:rgb(255,235,156)"><b><span style="font-size:10pt;color:rgb(156,101,0)"></span></b><span style="font-size:10pt;color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]</span></p>
</div>
<div>
<div dir="ltr">cleared disks of controller-0 and reinstall.
<div>make controller-0 as the active controller (controller-1 was the active controller)</div>
<div>deleted and added stx-openstack but still failing the same pod.</div>
<div><br>
</div>
<div><a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808170/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPMoCOLV4$" target="_blank">https://paste.opendev.org/show/808170/</a><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Aug 17, 2021 at 6:30 PM open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Douglas,
<div><br>
<div>Both pods have similar disk usage and <span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:12px">4.4GB remaining space out of 4.8GB in /var/lib/mysql directory.</span></div>
<div><a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808142/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPIFJURAp$" target="_blank">https://paste.opendev.org/show/808142/</a><br>
</div>
<div><br>
</div>
<div>I have allocated 200GB for docker-lv</div>
</div>
<div>Regards,</div>
<div>Danishka</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Aug 17, 2021 at 5:38 PM Douglas Lopes Pereira <<a href="mailto:douglas.pereira@windriver.com" target="_blank">douglas.pereira@windriver.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi Danishka,
<div><br>
</div>
<div>can you confirm there is enough disk space in your controller for the MariaDB pod to recover?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Doug</div>
</div>
<br>
<div>
<div dir="ltr">On Tue, Aug 17, 2021 at 8:25 AM open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p></p>
<div style="border:1pt solid rgb(156,101,0);padding:2pt">
<p style="line-height:12pt;background:rgb(255,235,156)"><b><span style="font-size:10pt;color:rgb(156,101,0)"></span></b><span style="font-size:10pt;color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]<u></u><u></u></span></p>
</div>
<div>
<div dir="ltr">
<div><br>
</div>
<div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px">In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod.</span></div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px">This pod keeps rebooting and also stx-openstack applying keep fails each time.</span></div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px"><br>
</span></div>
<div>
<pre style="white-space:pre-wrap;line-height:16.25px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)">2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist</pre>
</div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px"> </span></div>
<div>[1] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$" target="_blank">https://paste.opendev.org/show/808124/</a><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px"><br>
</span></div>
<div>[2] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$" target="_blank">https://paste.opendev.org/show/808125/</a><br>
</div>
<div>[3] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$" target="_blank">https://paste.opendev.org/show/808126/</a><br>
</div>
<div>[4] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$" target="_blank">https://paste.opendev.org/show/808127/</a></div>
<div>[5] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$" target="_blank">https://paste.opendev.org/show/808128/</a>
<div style="outline:none;padding:10px 0px;width:22px;margin:2px 0px 0px"><br>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Starlingx-discuss mailing list<br>
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a><br>
<a href="https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPNYQ2kIe$" rel="noreferrer" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote></div></div>