[Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.

[Peters, Matt]

Hi Erich,
We are introducing a new application (stx-monitor) that will include the Open Source Elastic (https://www.elastic.co/) set of monitoring services that may be leveraged for this activity.  The Elastic components include Elasticsearch, Filebeat, Metricbeat, Logstash and Kibana.  Specific monitor of key logs and metrics can be performed via Elastic.  It is also K8s cluster aware, and provides monitoring of both the infrastructure and pods/containers.

In general, custom system level monitoring is integrated into collectd for in-service monitoring of the system.  It is responsible for raising system alarms and events through a continuous monitoring of the hosts and services.  The service manager (SM) will monitor K8s master services and report on that health independently and raise alarms and events as necessary (you don't need to look at sm-dump).

I recommend leveraging these existing components for any gaps that may be identified in the system level health monitoring.

-Matt

On 2019-05-14, 8:18 PM, "Cordoba Malibran, Erich" <erich.cordoba.malibran at intel.com> wrote:

    Hi,
    
    After some time debugging issues in StarlingX I think it would be
    helpful to have a tool (if there isn't one yet) to get a diagnostic of
    the health of the system. Something that an user, with a limited
    knowledge of the system like me, can run and get the status of the
    different components. 
    
    
    I would like to start writing some code for this and I have a general
    idea on which components to check.
    
    As an initial approach I plan to: 
    
      - Check that configuration has been applied without errors: I plan to
    check for puppet/ansible logs and look for errors there.
      - Check for services up and running: Look at the output of sm-dump.
      - Verify the health of k8s: Checking that no pods are in Init state.
      - Check connectivity with additional nodes: To check that nodes
    booted and has network access.
      - Not sure yet how to check storage status beyond "ceph -s"
    
    I know this might be a naive approach and I'll appreciate any guidance
    to get in the correct path.
    
    Thank you in advance.
    
    -Erich
    
    _______________________________________________
    Starlingx-discuss mailing list
    Starlingx-discuss at lists.starlingx.io
    http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss