[Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.

Thu May 23 11:35:07 UTC 2019

Also note that we already have a CLI that checks some basic system health (host status, ceph health and alarms). This CLI could be extended if there were additional health items we wanted to check.

[wrsroot at controller-0 ~(keystone_admin)]$ system health-query
System Health:
All hosts are provisioned: [OK]
All hosts are unlocked/enabled: [OK]
All hosts have current configurations: [OK]
All hosts are patch current: [OK]
Ceph Storage Healthy: [OK]
No alarms: [OK]

Bart

-----Original Message-----
From: Poncea, Ovidiu [mailto:Ovidiu.Poncea at windriver.com] 
Sent: May 23, 2019 5:42 AM
To: Victor Rodriguez; Bailey, Henry Albert (Al)
Cc: Cordoba Malibran, Erich; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.

Hi Victor,

The 3 questions are indeed very important! And they indeed should be asked permanently.  As a starting point it's worth looking first at StarlingX own Fault Management components. You can start from there and synthesize a set of conditions, per domain, that are a minimum requirement. 

Currently, if memory serves me, sanity check fm alarm-list for any error condition (so we do check this) but we don't have a custom/external set of scripts to test for this (test teams should confirm this). Of course this may not be enough (what happens if fm itself is broken, or there is a bug and we don't report an error condition?).

So, as a starting point check /etc/fm/events.yaml for defined error conditions then look at the collect tool for command used... then ask each domain what is the best approach.

Re using external tools. I believe this may be overkill. A set of custom tests that use our own commands (e.g. parse ceph -s, ceph osd tree, ceph df) should be enough.

Ovidiu
________________________________________
From: Victor Rodriguez [vm.rod25 at gmail.com]
Sent: Wednesday, May 15, 2019 5:38 PM
To: Bailey, Henry Albert (Al)
Cc: Poncea, Ovidiu; Cordoba Malibran, Erich; starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.

Is a great idea

Actually, I would like  this to be run as part of the sanity,

In my personal opinion we can start with these statements:

1) What does health means for our system? Define what is the meaning
of a healthy system first
2) What components should be up and running before starting to use a starling x
3) What open source tools exist to check the health of these systems,
can we just pick a minimum set of the tests from these projects

Once we have a clear picture of the full system and what test is going
to be performed, many of us will be very happy to add ideas on it

Regards

Victor Rodriguez

On Wed, May 15, 2019 at 9:16 AM Bailey, Henry Albert (Al)
<Al.Bailey at windriver.com> wrote:
>
> Some of these tools may help.  I don't know which ones are still operational.
>
> https://opendev.org/starlingx/integ/src/branch/master/tools
>
> Al
>
> -----Original Message-----
> From: Poncea, Ovidiu [mailto:Ovidiu.Poncea at windriver.com]
> Sent: Wednesday, May 15, 2019 4:58 AM
> To: Cordoba Malibran, Erich; starlingx-discuss at lists.starlingx.io
> Subject: Re: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.
>
> Hi Eric, nice idea! I can help with ceph status beyond "ceph -s" once you get there :)
>
> Ovidiu
>
> PS. Check "fm alarm list" first
>
> ________________________________________
> From: Cordoba Malibran, Erich [erich.cordoba.malibran at intel.com]
> Sent: Wednesday, May 15, 2019 3:18 AM
> To: starlingx-discuss at lists.starlingx.io
> Subject: [Starlingx-discuss] Diagnostic metrics of a healthy StarlingX  system.
>
> Hi,
>
> After some time debugging issues in StarlingX I think it would be
> helpful to have a tool (if there isn't one yet) to get a diagnostic of
> the health of the system. Something that an user, with a limited
> knowledge of the system like me, can run and get the status of the
> different components.
>
>
> I would like to start writing some code for this and I have a general
> idea on which components to check.
>
> As an initial approach I plan to:
>
>   - Check that configuration has been applied without errors: I plan to
> check for puppet/ansible logs and look for errors there.
>   - Check for services up and running: Look at the output of sm-dump.
>   - Verify the health of k8s: Checking that no pods are in Init state.
>   - Check connectivity with additional nodes: To check that nodes
> booted and has network access.
>   - Not sure yet how to check storage status beyond "ceph -s"
>
> I know this might be a naive approach and I'll appreciate any guidance
> to get in the correct path.
>
> Thank you in advance.
>
> -Erich
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss