Hi Thiago, Please find my inline comments. On Wed, Aug 18, 2021 at 8:28 PM Brito, Thiago <Thiago.Brito@windriver.com> wrote:
Hi Danishka,
Looking at your logs it seems like the missing table 'mysql.servers' is preventing the system to come up. Would you please raise a LP with those logs so we can dig deeper into this problem? It would be useful to have the full logs from these attempts.
Sure, I will raise a LP.
Once you do that, you might wanna try to "application-remove" stx-openstack and check if the PVC of mariadb is still on the system with "kubectl get pvc -n openstack". If that is the case, the app is probably trying to reuse the old "broken" database. Deleting that pvc and appling stx-openstack again should get your system up.
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pvc -n openstack No resources found in openstack namespace. [sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -n openstack -o wide No resources found in openstack namespace. https://paste.opendev.org/show/808178/
Thiago ------------------------------ *From:* open infra <openinfradn@gmail.com> *Sent:* Wednesday, August 18, 2021 11:13 AM *To:* Pereira, Douglas <Douglas.Pereira@windriver.com>; starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Sun, Austin <austin.sun@intel.com> *Subject:* Re: [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing
[Please note: This e-mail is from an EXTERNAL e-mail address] cleared disks of controller-0 and reinstall. make controller-0 as the active controller (controller-1 was the active controller) deleted and added stx-openstack but still failing the same pod.
https://paste.opendev.org/show/808170/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808170/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPMoCOLV4$>
On Tue, Aug 17, 2021 at 6:30 PM open infra <openinfradn@gmail.com> wrote:
Hi Douglas,
Both pods have similar disk usage and 4.4GB remaining space out of 4.8GB in /var/lib/mysql directory. https://paste.opendev.org/show/808142/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808142/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPIFJURAp$>
I have allocated 200GB for docker-lv Regards, Danishka
On Tue, Aug 17, 2021 at 5:38 PM Douglas Lopes Pereira < douglas.pereira@windriver.com> wrote:
Hi Danishka,
can you confirm there is enough disk space in your controller for the MariaDB pod to recover?
Regards, Doug
On Tue, Aug 17, 2021 at 8:25 AM open infra <openinfradn@gmail.com> wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod. This pod keeps rebooting and also stx-openstack applying keep fails each time.
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
[1] https://paste.opendev.org/show/808124/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$> [2] https://paste.opendev.org/show/808125/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$> [3] https://paste.opendev.org/show/808126/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$> [4] https://paste.opendev.org/show/808127/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$> [5] https://paste.opendev.org/show/808128/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$>
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss <https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPNYQ2kIe$>