[Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.

[Hernandez Gonzalez, Fernando]

Sounds like a really good idea :-) 

For ceph storage, what about if we start with:

a) "ceph -s" and check no WARNs showed. 
b) Then "ceph osd tree" and check all osds are up (no down should be shown check below example just happened to me)
   controller-0:~$ ceph osd tree
   ID  CLASS WEIGHT  TYPE NAME              STATUS REWEIGHT PRI-AFF
    -1       1.74194 root storage-tier
    -3       1.30646     chassis group-0
    -4       0.43549         host storage-0
     0   ssd 0.43549             osd.0          up  1.00000 1.00000
    -5       0.87097         host storage-1
     1   ssd 0.43549             osd.1          up  1.00000 1.00000
     2   ssd 0.43549             osd.2          up  1.00000 1.00000
   -10       0.43549     chassis group-1
    -9       0.43549         host storage-2
     3   ssd 0.43549             osd.3        down  1.00000 1.00000   
   controller-0:~$ date 
   Tue May 14 18:25:59 CDT 2019
c) and after the tool can check services running in all the nodes "ceph-mon, ceph-mgr, ceph-msgr"

That will help for ceph storage diagnostic.

Regards,

Fernando Hernandez Gonzalez
Cloud Software Engineer

Avenida del Bosque #1001 Col, El Bajío
 Zapopan, Jalisco MX, 45019
 ____________________________________
Office: +52.33.16.45.01.34   inet 86450134

-----Original Message-----
From: Cordoba Malibran, Erich [mailto:erich.cordoba.malibran at intel.com] 
Sent: Tuesday, May 14, 2019 7:18 PM
To: starlingx-discuss at lists.starlingx.io
Subject: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.

Hi,

After some time debugging issues in StarlingX I think it would be helpful to have a tool (if there isn't one yet) to get a diagnostic of the health of the system. Something that an user, with a limited knowledge of the system like me, can run and get the status of the different components. 


I would like to start writing some code for this and I have a general idea on which components to check.

As an initial approach I plan to: 

  - Check that configuration has been applied without errors: I plan to check for puppet/ansible logs and look for errors there.
  - Check for services up and running: Look at the output of sm-dump.
  - Verify the health of k8s: Checking that no pods are in Init state.
  - Check connectivity with additional nodes: To check that nodes booted and has network access.
  - Not sure yet how to check storage status beyond "ceph -s"

I know this might be a naive approach and I'll appreciate any guidance to get in the correct path.

Thank you in advance.

-Erich

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss