Hi Thiago,

Please find my inline comments.

On Wed, Aug 18, 2021 at 8:28 PM Brito, Thiago <Thiago.Brito@windriver.com> wrote:
Hi Danishka,

Looking at your logs it seems like the missing table 'mysql.servers' is preventing the system to come up. Would you please raise a LP with those logs so we can dig deeper into this problem? It would be useful to have the full logs from these attempts.


Sure, I will raise a LP.
 
Once you do that, you might wanna try to "application-remove" stx-openstack and check if the PVC of mariadb is still on the system with "kubectl get pvc -n openstack". If that is the case, the app is probably trying to reuse the old "broken" database. Deleting that pvc and appling stx-openstack again should get your system up.

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pvc -n openstack

No resources found in openstack namespace.

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n openstack -o wide

No resources found in openstack namespace.



https://paste.opendev.org/show/808178/
 

Thiago

From: open infra <openinfradn@gmail.com>
Sent: Wednesday, August 18, 2021 11:13 AM
To: Pereira, Douglas <Douglas.Pereira@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Sun, Austin <austin.sun@intel.com>
Subject: Re: [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing
 

[Please note: This e-mail is from an EXTERNAL e-mail address]

cleared disks of controller-0 and reinstall.
make controller-0 as the active controller (controller-1 was the active controller)
deleted and added stx-openstack  but still failing the same pod.


On Tue, Aug 17, 2021 at 6:30 PM open infra <openinfradn@gmail.com> wrote:
Hi Douglas,

Both pods have similar disk usage and 4.4GB remaining space out of 4.8GB in /var/lib/mysql directory.

I have allocated 200GB for docker-lv
Regards,
Danishka

On Tue, Aug 17, 2021 at 5:38 PM Douglas Lopes Pereira <douglas.pereira@windriver.com> wrote:
Hi Danishka,

can you confirm there is enough disk space in your controller for the MariaDB pod to recover?

Regards,
Doug

On Tue, Aug 17, 2021 at 8:25 AM open infra <openinfradn@gmail.com> wrote:

[Please note: This e-mail is from an EXTERNAL e-mail address]


In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod.
This pod keeps rebooting and also stx-openstack applying keep fails each time.

2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
 
_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss@lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss