[Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing

[Brito, Thiago]

Hi Danishka,

Looking at your logs it seems like the missing table 'mysql.servers' is preventing the system to come up. Would you please raise a LP with those logs so we can dig deeper into this problem? It would be useful to have the full logs from these attempts.

Once you do that, you might wanna try to "application-remove" stx-openstack and check if the PVC of mariadb is still on the system with "kubectl get pvc -n openstack". If that is the case, the app is probably trying to reuse the old "broken" database. Deleting that pvc and appling stx-openstack again should get your system up.

Thiago
________________________________
From: open infra <openinfradn at gmail.com>
Sent: Wednesday, August 18, 2021 11:13 AM
To: Pereira, Douglas <Douglas.Pereira at windriver.com>; starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>; Sun, Austin <austin.sun at intel.com>
Subject: Re: [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing


[Please note: This e-mail is from an EXTERNAL e-mail address]

cleared disks of controller-0 and reinstall.
make controller-0 as the active controller (controller-1 was the active controller)
deleted and added stx-openstack  but still failing the same pod.

https://paste.opendev.org/show/808170/<https://urldefense.com/v3/__https://paste.opendev.org/show/808170/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPMoCOLV4$>

On Tue, Aug 17, 2021 at 6:30 PM open infra <openinfradn at gmail.com<mailto:openinfradn at gmail.com>> wrote:
Hi Douglas,

Both pods have similar disk usage and 4.4GB remaining space out of 4.8GB in /var/lib/mysql directory.
https://paste.opendev.org/show/808142/<https://urldefense.com/v3/__https://paste.opendev.org/show/808142/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPIFJURAp$>

I have allocated 200GB for docker-lv
Regards,
Danishka

On Tue, Aug 17, 2021 at 5:38 PM Douglas Lopes Pereira <douglas.pereira at windriver.com<mailto:douglas.pereira at windriver.com>> wrote:
Hi Danishka,

can you confirm there is enough disk space in your controller for the MariaDB pod to recover?

Regards,
Doug

On Tue, Aug 17, 2021 at 8:25 AM open infra <openinfradn at gmail.com<mailto:openinfradn at gmail.com>> wrote:

[Please note: This e-mail is from an EXTERNAL e-mail address]

In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod.
This pod keeps rebooting and also stx-openstack applying keep fails each time.


2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist


[1] https://paste.opendev.org/show/808124/<https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$>
[2] https://paste.opendev.org/show/808125/<https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$>
[3] https://paste.opendev.org/show/808126/<https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$>
[4] https://paste.opendev.org/show/808127/<https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$>
[5] https://paste.opendev.org/show/808128/<https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$>

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io<mailto:Starlingx-discuss at lists.starlingx.io>
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss<https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPNYQ2kIe$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210818/311a340d/attachment-0001.html>