Hi Erich, We are introducing a new application (stx-monitor) that will include the Open Source Elastic (https://www.elastic.co/) set of monitoring services that may be leveraged for this activity. The Elastic components include Elasticsearch, Filebeat, Metricbeat, Logstash and Kibana. Specific monitor of key logs and metrics can be performed via Elastic. It is also K8s cluster aware, and provides monitoring of both the infrastructure and pods/containers. In general, custom system level monitoring is integrated into collectd for in-service monitoring of the system. It is responsible for raising system alarms and events through a continuous monitoring of the hosts and services. The service manager (SM) will monitor K8s master services and report on that health independently and raise alarms and events as necessary (you don't need to look at sm-dump). I recommend leveraging these existing components for any gaps that may be identified in the system level health monitoring. -Matt On 2019-05-14, 8:18 PM, "Cordoba Malibran, Erich" <erich.cordoba.malibran@intel.com> wrote: Hi, After some time debugging issues in StarlingX I think it would be helpful to have a tool (if there isn't one yet) to get a diagnostic of the health of the system. Something that an user, with a limited knowledge of the system like me, can run and get the status of the different components. I would like to start writing some code for this and I have a general idea on which components to check. As an initial approach I plan to: - Check that configuration has been applied without errors: I plan to check for puppet/ansible logs and look for errors there. - Check for services up and running: Look at the output of sm-dump. - Verify the health of k8s: Checking that no pods are in Init state. - Check connectivity with additional nodes: To check that nodes booted and has network access. - Not sure yet how to check storage status beyond "ceph -s" I know this might be a naive approach and I'll appreciate any guidance to get in the correct path. Thank you in advance. -Erich _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss