Sounds like a really good idea :-) For ceph storage, what about if we start with: a) "ceph -s" and check no WARNs showed. b) Then "ceph osd tree" and check all osds are up (no down should be shown check below example just happened to me) controller-0:~$ ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 1.74194 root storage-tier -3 1.30646 chassis group-0 -4 0.43549 host storage-0 0 ssd 0.43549 osd.0 up 1.00000 1.00000 -5 0.87097 host storage-1 1 ssd 0.43549 osd.1 up 1.00000 1.00000 2 ssd 0.43549 osd.2 up 1.00000 1.00000 -10 0.43549 chassis group-1 -9 0.43549 host storage-2 3 ssd 0.43549 osd.3 down 1.00000 1.00000 controller-0:~$ date Tue May 14 18:25:59 CDT 2019 c) and after the tool can check services running in all the nodes "ceph-mon, ceph-mgr, ceph-msgr" That will help for ceph storage diagnostic. Regards, Fernando Hernandez Gonzalez Cloud Software Engineer Avenida del Bosque #1001 Col, El Bajío Zapopan, Jalisco MX, 45019 ____________________________________ Office: +52.33.16.45.01.34 inet 86450134 -----Original Message----- From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran@intel.com] Sent: Tuesday, May 14, 2019 7:18 PM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system. Hi, After some time debugging issues in StarlingX I think it would be helpful to have a tool (if there isn't one yet) to get a diagnostic of the health of the system. Something that an user, with a limited knowledge of the system like me, can run and get the status of the different components. I would like to start writing some code for this and I have a general idea on which components to check. As an initial approach I plan to: - Check that configuration has been applied without errors: I plan to check for puppet/ansible logs and look for errors there. - Check for services up and running: Look at the output of sm-dump. - Verify the health of k8s: Checking that no pods are in Init state. - Check connectivity with additional nodes: To check that nodes booted and has network access. - Not sure yet how to check storage status beyond "ceph -s" I know this might be a naive approach and I'll appreciate any guidance to get in the correct path. Thank you in advance. -Erich _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss