<div dir="ltr">cleared disks of controller-0 and reinstall.<div>make controller-0 as the active controller (controller-1 was the active controller)</div><div>deleted and added stx-openstack  but still failing the same pod.</div><div><br></div><div><a href="https://paste.opendev.org/show/808170/">https://paste.opendev.org/show/808170/</a><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 17, 2021 at 6:30 PM open infra <<a href="mailto:openinfradn@gmail.com">openinfradn@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Douglas,<div><br><div>Both pods have similar disk usage and <span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:12px">4.4GB remaining space out of 4.8GB in /var/lib/mysql directory.</span></div>





<div><a href="https://paste.opendev.org/show/808142/" target="_blank">https://paste.opendev.org/show/808142/</a><br></div><div><br></div><div>I have allocated 200GB for docker-lv</div></div><div>Regards,</div><div>Danishka</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 17, 2021 at 5:38 PM Douglas Lopes Pereira <<a href="mailto:douglas.pereira@windriver.com" target="_blank">douglas.pereira@windriver.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Danishka,<div><br></div><div>can you confirm there is enough disk space in your controller for the MariaDB pod to recover?</div><div><br></div><div>Regards,</div><div>Doug</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 17, 2021 at 8:25 AM open infra <<a href="mailto:openinfradn@gmail.com" target="_blank">openinfradn@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">


<div>
<p></p>
<div style="border:1pt solid rgb(156,101,0);padding:2pt">
<p class="MsoNormal" style="line-height:12pt;background:rgb(255,235,156)"><b><span style="font-size:10pt;color:rgb(156,101,0)"></span></b><span style="font-size:10pt;color:red">[Please note: This e-mail is from an EXTERNAL e-mail address]<u></u><u></u></span></p>
</div>
<div>
<div dir="ltr">
<div><br>
</div>
<div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px">In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod.</span></div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px">This pod keeps rebooting and also stx-openstack applying keep fails each time.</span></div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px"><br>
</span></div>
<div>
<pre style="white-space:pre-wrap;line-height:16.25px;margin-top:0px;margin-bottom:0px;padding:5px 0px;font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px;color:rgb(0,0,0)">2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist</pre>
</div>
<div><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px"> </span></div>
<div>[1] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$" target="_blank">https://paste.opendev.org/show/808124/</a><span style="color:rgb(0,0,0);font-family:"Bitstream Vera Sans Mono",monospace;font-size:13px"><br>
</span></div>
<div>[2] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$" target="_blank">https://paste.opendev.org/show/808125/</a><br>
</div>
<div>[3] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$" target="_blank">https://paste.opendev.org/show/808126/</a><br>
</div>
<div>[4] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$" target="_blank">https://paste.opendev.org/show/808127/</a></div>
<div>[5] <a href="https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$" target="_blank">https://paste.opendev.org/show/808128/</a>
<div style="outline:none;padding:10px 0px;width:22px;margin:2px 0px 0px">
<br>
</div>
</div>
</div>
</div>
</div>
</div>

_______________________________________________<br>
Starlingx-discuss mailing list<br>
<a href="mailto:Starlingx-discuss@lists.starlingx.io" target="_blank">Starlingx-discuss@lists.starlingx.io</a><br>
<a href="http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss" rel="noreferrer" target="_blank">http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>