[Starlingx-discuss] Need a help to recover Openstack Inbox | specific pod is failing

[open infra]

Hi Thiago,

Please find my inline comments.

On Wed, Aug 18, 2021 at 8:28 PM Brito, Thiago <Thiago.Brito at windriver.com>
wrote:

> Hi Danishka,
>
> Looking at your logs it seems like the missing table 'mysql.servers' is
> preventing the system to come up. Would you please raise a LP with those
> logs so we can dig deeper into this problem? It would be useful to have the
> full logs from these attempts.
>
>
Sure, I will raise a LP.


> Once you do that, you might wanna try to "application-remove"
> stx-openstack and check if the PVC of mariadb is still on the system with
> "kubectl get pvc -n openstack". If that is the case, the app is probably
> trying to reuse the old "broken" database. Deleting that pvc and appling
> stx-openstack again should get your system up.
>

[sysadmin at controller-0 ~(keystone_admin)]$ kubectl get pvc -n openstack

No resources found in openstack namespace.

[sysadmin at controller-0 ~(keystone_admin)]$ kubectl get pods -n openstack -o
wide

No resources found in openstack namespace.



https://paste.opendev.org/show/808178/


>
> Thiago
> ------------------------------
> *From:* open infra <openinfradn at gmail.com>
> *Sent:* Wednesday, August 18, 2021 11:13 AM
> *To:* Pereira, Douglas <Douglas.Pereira at windriver.com>;
> starlingx-discuss at lists.starlingx.io <starlingx-discuss at lists.starlingx.io>;
> Sun, Austin <austin.sun at intel.com>
> *Subject:* Re: [Starlingx-discuss] Need a help to recover Openstack Inbox
> | specific pod is failing
>
>
> [Please note: This e-mail is from an EXTERNAL e-mail address]
> cleared disks of controller-0 and reinstall.
> make controller-0 as the active controller (controller-1 was the active
> controller)
> deleted and added stx-openstack  but still failing the same pod.
>
> https://paste.opendev.org/show/808170/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/808170/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPMoCOLV4$>
>
> On Tue, Aug 17, 2021 at 6:30 PM open infra <openinfradn at gmail.com> wrote:
>
> Hi Douglas,
>
> Both pods have similar disk usage and 4.4GB remaining space out of 4.8GB
> in /var/lib/mysql directory.
> https://paste.opendev.org/show/808142/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/808142/__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPIFJURAp$>
>
> I have allocated 200GB for docker-lv
> Regards,
> Danishka
>
> On Tue, Aug 17, 2021 at 5:38 PM Douglas Lopes Pereira <
> douglas.pereira at windriver.com> wrote:
>
> Hi Danishka,
>
> can you confirm there is enough disk space in your controller for the
> MariaDB pod to recover?
>
> Regards,
> Doug
>
> On Tue, Aug 17, 2021 at 8:25 AM open infra <openinfradn at gmail.com> wrote:
>
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> In the logs of the pod (mariadb-server-1) [4], I noticed issues related to
> mariadb tables but only in the failed pod.
> This pod keeps rebooting and also stx-openstack applying keep fails each
> time.
>
> 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
> 2021-08-17 00:19:12,237 - OpenStack-Helm Mariadb - INFO - Recovering wsrep position: 2021-08-17  0:19:10 140357388367552 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
>
>
> [1] https://paste.opendev.org/show/808124/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/808124/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdRGCXd6I$>
> [2] https://paste.opendev.org/show/808125/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/808125/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdf-z9uw4$>
> [3] https://paste.opendev.org/show/808126/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/808126/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdZf9xphM$>
> [4] https://paste.opendev.org/show/808127/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/808127/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdPrJSRFQ$>
> [5] https://paste.opendev.org/show/808128/
> <https://urldefense.com/v3/__https://paste.opendev.org/show/808128/__;!!AjveYdw8EvQ!JTJiAZMdjI3CRt_GTKgeYIyeAHTSZRk_ujxNaClvLz0Z5FqCTpWQKA6u1MwSZmVdiCHuuC8$>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> <https://urldefense.com/v3/__http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!NqBlJfyRLtS3pm9zbw75s_9goG0rM7I7DdsryX-_V0iV641C1S1lsDbwoOPxPNYQ2kIe$>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20210818/ebcb419f/attachment.html>