[Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing
Hi, Openstack was failed and noticed that mariadb-server-1 pod is failing. openstack mariadb-server-0 0/1 Running openstack mariadb-server-1 0/1 CrashLoopBackOff I managed to get logs of each pod. mariadb-server-0 log: https://paste.opendev.org/show/808070/ and https://paste.opendev.org/show/808071/ mariadb-server-1 log: https://paste.opendev.org/show/808067/ I highly appreciate if I can get a hint or steps to resolve this issue. Regards, Danishka
Hi Danishka: Mariadb-server-0 is new leading , but I did find any clue of server-1 crash info from log [1]. Are controllers available and any alarms ? [1] https://paste.opendev.org/show/808067/ Thanks. BR Austin Sun. From: open infra <openinfradn@gmail.com> Sent: Saturday, August 14, 2021 2:27 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing Hi, Openstack was failed and noticed that mariadb-server-1 pod is failing. openstack mariadb-server-0 0/1 Running openstack mariadb-server-1 0/1 CrashLoopBackOff I managed to get logs of each pod. mariadb-server-0 log: https://paste.opendev.org/show/808070/ and https://paste.opendev.org/show/808071/ mariadb-server-1 log: https://paste.opendev.org/show/808067/ I highly appreciate if I can get a hint or steps to resolve this issue. Regards, Danishka
Hi Austin, On Sun, Aug 15, 2021 at 7:04 AM Sun, Austin <austin.sun@intel.com> wrote:
Hi Danishka:
Mariadb-server-0 is new leading , but I did find any clue of server-1 crash info from log [1].
Are controllers available and any alarms ?
Both controllers active and no alarms against controllers but 'Application Apply Failure' (stx -openstack) I have deleted all MariaDB pods, expecting new pods to be created by Kubernetes. But same result.
[1] https://paste.opendev.org/show/808067/
Thanks. BR Austin Sun.
*From:* open infra <openinfradn@gmail.com> *Sent:* Saturday, August 14, 2021 2:27 PM *To:* starlingx-discuss@lists.starlingx.io *Subject:* [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing
Hi,
Openstack was failed and noticed that mariadb-server-1 pod is failing.
openstack mariadb-server-0 0/1 Running
openstack mariadb-server-1 0/1 CrashLoopBackOff
I managed to get logs of each pod.
mariadb-server-0 log:
https://paste.opendev.org/show/808070/
and
https://paste.opendev.org/show/808071/
mariadb-server-1 log:
https://paste.opendev.org/show/808067/
I highly appreciate if I can get a hint or steps to resolve this issue.
Regards,
Danishka
In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod. This pod keeps rebooting and also stx-openstack applying keep fails each time. 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist [1] https://paste.opendev.org/show/808124/ [2] https://paste.opendev.org/show/808125/ [3] https://paste.opendev.org/show/808126/ [4] https://paste.opendev.org/show/808127/ [5] https://paste.opendev.org/show/808128/
Hi Danishka, can you confirm there is enough disk space in your controller for the MariaDB pod to recover? Regards, Doug On Tue, Aug 17, 2021 at 8:25 AM open infra <openinfradn@gmail.com> wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod. This pod keeps rebooting and also stx-openstack applying keep fails each time.
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
[1] https://paste.opendev.org/show/808124/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$> [2] https://paste.opendev.org/show/808125/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$> [3] https://paste.opendev.org/show/808126/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$> [4] https://paste.opendev.org/show/808127/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$> [5] https://paste.opendev.org/show/808128/ <https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$>
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (3)
-
Douglas Lopes Pereira
-
open infra
-
Sun, Austin