Thanks Erich for taking care of a healthy StarlingX System. Matt, You are bringing a very important topic which we would like to address now, with the only objective to take advantage of your experience and the work you have completed.
We are introducing a new application (stx-monitor) that will include the Open Source Elastic (https://www.elastic.co/) set of monitoring services that may be leveraged for this activity. The Elastic components include Elasticsearch, Filebeat, Metricbeat, Logstash and Kibana. Specific monitor of key logs and metrics can be performed via Elastic. It is also K8s cluster aware, and provides monitoring of both the infrastructure and pods/containers.
From our side, we are working to enable a set of tools around metrics / performance, here you have the StarlingX Metrics board: https://storyboard.openstack.org/#!/board/145 Which contains today 2 high level stories: - Footprint Metrics: CPU, Memory, Storage v0.1 + Alerts v0.3 - Performance Metrics: Include OPNFV We are focusing today in Footprint Collection System which is basically the leverage of the following existing projects: - System Server Side - Prometheus - Grafana - Cluster Node Side - Node Exporter
In general, custom system level monitoring is integrated into collectd for in- service monitoring of the system. It is responsible for raising system alarms and events through a continuous monitoring of the hosts and services. The service manager (SM) will monitor K8s master services and report on that health independently and raise alarms and events as necessary (you don't need to look at sm-dump).
I recommend leveraging these existing components for any gaps that may be identified in the system level health monitoring.
Yes, let us ask you these question and your support, based in your experience: - Are we working in parallel with 2 different approaches for the same task? - Are they 2 different tasks that at some point could be integrated together? - Can we take a look at the development of stx-monitor to understand more about its architecture? - Can we have some time in our community meetings to get more into the details of stx-monitor and find out possible collaboration activities? Best Regards StarlingX Mexico Team