[Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.
Hernandez Gonzalez, Fernando
fernando.hernandez.gonzalez at intel.com
Wed May 15 05:36:08 UTC 2019
Sounds like a really good idea :-)
For ceph storage, what about if we start with:
a) "ceph -s" and check no WARNs showed.
b) Then "ceph osd tree" and check all osds are up (no down should be shown check below example just happened to me)
controller-0:~$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 1.74194 root storage-tier
-3 1.30646 chassis group-0
-4 0.43549 host storage-0
0 ssd 0.43549 osd.0 up 1.00000 1.00000
-5 0.87097 host storage-1
1 ssd 0.43549 osd.1 up 1.00000 1.00000
2 ssd 0.43549 osd.2 up 1.00000 1.00000
-10 0.43549 chassis group-1
-9 0.43549 host storage-2
3 ssd 0.43549 osd.3 down 1.00000 1.00000
Tue May 14 18:25:59 CDT 2019
c) and after the tool can check services running in all the nodes "ceph-mon, ceph-mgr, ceph-msgr"
That will help for ceph storage diagnostic.
Fernando Hernandez Gonzalez
Cloud Software Engineer
Avenida del Bosque #1001 Col, El Bajío
Zapopan, Jalisco MX, 45019
Office: +184.108.40.206.01.34 inet 86450134
From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran at intel.com]
Sent: Tuesday, May 14, 2019 7:18 PM
To: starlingx-discuss at lists.starlingx.io
Subject: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.
After some time debugging issues in StarlingX I think it would be helpful to have a tool (if there isn't one yet) to get a diagnostic of the health of the system. Something that an user, with a limited knowledge of the system like me, can run and get the status of the different components.
I would like to start writing some code for this and I have a general idea on which components to check.
As an initial approach I plan to:
- Check that configuration has been applied without errors: I plan to check for puppet/ansible logs and look for errors there.
- Check for services up and running: Look at the output of sm-dump.
- Verify the health of k8s: Checking that no pods are in Init state.
- Check connectivity with additional nodes: To check that nodes booted and has network access.
- Not sure yet how to check storage status beyond "ceph -s"
I know this might be a naive approach and I'll appreciate any guidance to get in the correct path.
Thank you in advance.
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
More information about the Starlingx-discuss