Hi Victor, The 3 questions are indeed very important! And they indeed should be asked permanently. As a starting point it's worth looking first at StarlingX own Fault Management components. You can start from there and synthesize a set of conditions, per domain, that are a minimum requirement. Currently, if memory serves me, sanity check fm alarm-list for any error condition (so we do check this) but we don't have a custom/external set of scripts to test for this (test teams should confirm this). Of course this may not be enough (what happens if fm itself is broken, or there is a bug and we don't report an error condition?). So, as a starting point check /etc/fm/events.yaml for defined error conditions then look at the collect tool for command used... then ask each domain what is the best approach. Re using external tools. I believe this may be overkill. A set of custom tests that use our own commands (e.g. parse ceph -s, ceph osd tree, ceph df) should be enough. Ovidiu ________________________________________ From: Victor Rodriguez [vm.rod25@gmail.com] Sent: Wednesday, May 15, 2019 5:38 PM To: Bailey, Henry Albert (Al) Cc: Poncea, Ovidiu; Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system. Is a great idea Actually, I would like this to be run as part of the sanity, In my personal opinion we can start with these statements: 1) What does health means for our system? Define what is the meaning of a healthy system first 2) What components should be up and running before starting to use a starling x 3) What open source tools exist to check the health of these systems, can we just pick a minimum set of the tests from these projects Once we have a clear picture of the full system and what test is going to be performed, many of us will be very happy to add ideas on it Regards Victor Rodriguez On Wed, May 15, 2019 at 9:16 AM Bailey, Henry Albert (Al) <Al.Bailey@windriver.com> wrote:
Some of these tools may help. I don't know which ones are still operational.
https://opendev.org/starlingx/integ/src/branch/master/tools
Al
-----Original Message----- From: Poncea, Ovidiu [mailto:Ovidiu.Poncea@windriver.com] Sent: Wednesday, May 15, 2019 4:58 AM To: Cordoba Malibran, Erich; starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.
Hi Eric, nice idea! I can help with ceph status beyond "ceph -s" once you get there :)
Ovidiu
PS. Check "fm alarm list" first
________________________________________ From: Cordoba Malibran, Erich [erich.cordoba.malibran@intel.com] Sent: Wednesday, May 15, 2019 3:18 AM To: starlingx-discuss@lists.starlingx.io Subject: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.
Hi,
After some time debugging issues in StarlingX I think it would be helpful to have a tool (if there isn't one yet) to get a diagnostic of the health of the system. Something that an user, with a limited knowledge of the system like me, can run and get the status of the different components.
I would like to start writing some code for this and I have a general idea on which components to check.
As an initial approach I plan to:
- Check that configuration has been applied without errors: I plan to check for puppet/ansible logs and look for errors there. - Check for services up and running: Look at the output of sm-dump. - Verify the health of k8s: Checking that no pods are in Init state. - Check connectivity with additional nodes: To check that nodes booted and has network access. - Not sure yet how to check storage status beyond "ceph -s"
I know this might be a naive approach and I'll appreciate any guidance to get in the correct path.
Thank you in advance.
-Erich
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss