<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Danishka,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Looking at your logs it seems like the missing table <span style="font-family: "Bitstream Vera Sans Mono", monospace; font-size: 13px; color: rgb(0, 0, 0);">'mysql.servers' is preventing the system to come up. Would you please raise a LP with those logs so
we can dig deeper into this problem? It would be useful to have the full logs from these attempts.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Bitstream Vera Sans Mono", monospace; font-size: 13px; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Bitstream Vera Sans Mono", monospace; font-size: 13px; color: rgb(0, 0, 0);">Once you do that, you might wanna try to "application-remove" stx-openstack and check if the PVC of mariadb is still on the system with "kubectl get pvc
-n openstack". If that is the case, the app is probably trying to reuse the old "broken" database. Deleting that pvc and appling stx-openstack again should get your system up.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Bitstream Vera Sans Mono", monospace; font-size: 13px; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Bitstream Vera Sans Mono", monospace; font-size: 13px; color: rgb(0, 0, 0);">Thiago</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> open infra <openinfradn@gmail.com><br>
<b>Sent:</b> Wednesday, August 18, 2021 11:13 AM<br>
<b>To:</b> Pereira, Douglas <Douglas.Pereira@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Sun, Austin <austin.sun@intel.com><br>
<b>Subject:</b> Re: [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing</font>
<div> </div>
</div>
<div>
<p></p>
<div style="border:solid #9C6500 1.0pt; padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="x_MsoNormal" style="line-height:12.0pt; background:#FFEB9C"><b><span style="font-size:10.0pt; color:#9C6500"></span></b><span style="font-size:10.0pt; color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]</span></p>
</div>
<div>
<div dir="ltr">cleared disks of controller-0 and reinstall.
<div>make controller-0 as the active controller (controller-1 was the active controller)</div>
<div>deleted and added stx-openstack but still failing the same pod.</div>
<div><br>
</div>
<div><a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808170/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPMoCOLV4$">https://paste.opendev.org/show/808170/</a><br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Aug 17, 2021 at 6:30 PM open infra <<a href="mailto:openinfradn@gmail.com">openinfradn@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi Douglas,
<div><br>
<div>Both pods have similar disk usage and <span style="font-variant-ligatures:no-common-ligatures; color:rgb(0,0,0); font-family:Menlo; font-size:12px">4.4GB remaining space out of 4.8GB in /var/lib/mysql directory.</span></div>
<div><a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808142/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPIFJURAp$" target="_blank">https://paste.opendev.org/show/808142/</a><br>
</div>
<div><br>
</div>
<div>I have allocated 200GB for docker-lv</div>
</div>
<div>Regards,</div>
<div>Danishka</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Aug 17, 2021 at 5:38 PM Douglas Lopes Pereira <<a href="mailto:douglas.pereira@windriver.com" target="_blank">douglas.pereira@windriver.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">Hi Danishka,
<div><br>
</div>
<div>can you confirm there is enough disk space in your controller for the MariaDB pod to recover?</div>
<div><br>
</div>
<div>Regards,</div>
<div>Doug</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Tue, Aug 17, 2021 at 8:25 AM open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<p></p>
<div style="border:1pt solid rgb(156,101,0); padding:2pt">
<p class="x_MsoNormal" style="line-height:12pt; background:rgb(255,235,156)"><b><span style="font-size:10pt; color:rgb(156,101,0)"></span></b><span style="font-size:10pt; color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]<u></u><u></u></span></p>
</div>
<div>
<div dir="ltr">
<div><br>
</div>
<div>
<div><span style="color:rgb(0,0,0); font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px">In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod.</span></div>
<div><span style="color:rgb(0,0,0); font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px">This pod keeps rebooting and also stx-openstack applying keep fails each time.</span></div>
<div><span style="color:rgb(0,0,0); font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px"><br>
</span></div>
<div>
<pre style="white-space:pre-wrap; line-height:16.25px; margin-top:0px; margin-bottom:0px; padding:5px 0px; font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px; color:rgb(0,0,0)">2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist</pre>
</div>
<div><span style="color:rgb(0,0,0); font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px"> </span></div>
<div>[1] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$" target="_blank">https://paste.opendev.org/show/808124/</a><span style="color:rgb(0,0,0); font-family:"Bitstream Vera Sans Mono",monospace; font-size:13px"><br>
</span></div>
<div>[2] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$" target="_blank">https://paste.opendev.org/show/808125/</a><br>
</div>
<div>[3] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$" target="_blank">https://paste.opendev.org/show/808126/</a><br>
</div>
<div>[4] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$" target="_blank">https://paste.opendev.org/show/808127/</a></div>
<div>[5] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$" target="_blank">https://paste.opendev.org/show/808128/</a>
<div style="outline:none; padding:10px 0px; width:22px; margin:2px 0px 0px"><br>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Starlingx-discuss mailing list<br>
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a><br>
<a href="https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPNYQ2kIe$" rel="noreferrer" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>