[Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing
Hi,
Openstack was failed and noticed that mariadb-server-1 pod is failing.
openstack mariadb-server-0 0/1 Running
openstack mariadb-server-1 0/1 CrashLoopBackOff
I managed to get logs of each pod.
mariadb-server-0 log:
https://paste.opendev.org/show/808070/
and
https://paste.opendev.org/show/808071/
mariadb-server-1 log:
https://paste.opendev.org/show/808067/
I highly appreciate if I can get a hint or steps to resolve this issue.
Regards,
Danishka
Hi Danishka: Mariadb-server-0 is new leading , but I did find any clue of server-1 crash info from log [1].
Are controllers available and any alarms ?
[1] https://paste.opendev.org/show/808067/
Thanks. BR Austin Sun. From: open infra openinfradn@gmail.com Sent: Saturday, August 14, 2021 2:27 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing
Hi,
Openstack was failed and noticed that mariadb-server-1 pod is failing.
openstack mariadb-server-0 0/1 Running
openstack mariadb-server-1 0/1 CrashLoopBackOff
I managed to get logs of each pod.
mariadb-server-0 log:
https://paste.opendev.org/show/808070/
and
https://paste.opendev.org/show/808071/
mariadb-server-1 log:
https://paste.opendev.org/show/808067/
I highly appreciate if I can get a hint or steps to resolve this issue.
Regards,
Danishka
Hi Austin,
On Sun, Aug 15, 2021 at 7:04 AM Sun, Austin austin.sun@intel.com wrote:
Hi Danishka:
Mariadb-server-0 is new leading , but I did find any clue of server-1 crash info from log [1].
Are controllers available and any alarms ?
Both controllers active and no alarms against controllers but 'Application Apply Failure' (stx -openstack) I have deleted all MariaDB pods, expecting new pods to be created by Kubernetes. But same result.
[1] https://paste.opendev.org/show/808067/
Thanks. BR Austin Sun.
*From:* open infra openinfradn@gmail.com *Sent:* Saturday, August 14, 2021 2:27 PM *To:* starlingx-discuss@lists.starlingx.io *Subject:* [Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing
Hi,
Openstack was failed and noticed that mariadb-server-1 pod is failing.
openstack mariadb-server-0 0/1 Running
openstack mariadb-server-1 0/1 CrashLoopBackOff
I managed to get logs of each pod.
mariadb-server-0 log:
https://paste.opendev.org/show/808070/
and
https://paste.opendev.org/show/808071/
mariadb-server-1 log:
https://paste.opendev.org/show/808067/
I highly appreciate if I can get a hint or steps to resolve this issue.
Regards,
Danishka
In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod. This pod keeps rebooting and also stx-openstack applying keep fails each time.
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
[1] https://paste.opendev.org/show/808124/ [2] https://paste.opendev.org/show/808125/ [3] https://paste.opendev.org/show/808126/ [4] https://paste.opendev.org/show/808127/ [5] https://paste.opendev.org/show/808128/
Hi Danishka,
can you confirm there is enough disk space in your controller for the MariaDB pod to recover?
Regards, Doug
On Tue, Aug 17, 2021 at 8:25 AM open infra openinfradn@gmail.com wrote:
[Please note: This e-mail is from an EXTERNAL e-mail address]
In the logs of the pod (mariadb-server-1) [4], I noticed issues related to mariadb tables but only in the failed pod. This pod keeps rebooting and also stx-openstack applying keep fails each time.
2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17 0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
[1] https://paste.opendev.org/show/808124/ https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$ [2] https://paste.opendev.org/show/808125/ https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$ [3] https://paste.opendev.org/show/808126/ https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$ [4] https://paste.opendev.org/show/808127/ https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$ [5] https://paste.opendev.org/show/808128/ https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$
Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
participants (3)
-
Douglas Lopes Pereira
-
open infra
-
Sun, Austin