[Starlingx-discuss] Diagnostic metrics of a healthy StarlingX system.

[Peters, Matt]

Hi Abraham,
See inline.

On 2019-05-16, 12:10 PM, "Arce Moreno, Abraham" <abraham.arce.moreno at intel.com> wrote:

    Thanks Erich for taking care of a healthy StarlingX System.
    
    Matt,
    
    You are bringing a very important topic which we would like to address now, with the only objective to take advantage of your experience and the work you have completed.
    
    > We are introducing a new application (stx-monitor) that will include the Open
    > Source Elastic (https://www.elastic.co/) set of monitoring services that may be
    > leveraged for this activity.  The Elastic components include Elasticsearch,
    > Filebeat, Metricbeat, Logstash and Kibana.  Specific monitor of key logs and
    > metrics can be performed via Elastic.  It is also K8s cluster aware, and provides
    > monitoring of both the infrastructure and pods/containers.
    
    From our side, we are working to enable a set of tools around metrics / performance, here you have the StarlingX Metrics board:
    https://storyboard.openstack.org/#!/board/145

    Which contains today 2 high level stories:
    
    - Footprint Metrics: CPU, Memory, Storage v0.1 + Alerts v0.3
    - Performance Metrics: Include OPNFV
    
    We are focusing today in Footprint Collection System which is basically the leverage of the following existing projects:
    
    - System Server Side
      - Prometheus
      - Grafana
    - Cluster Node Side
      - Node Exporter

MP> I did see this story and some of the early commits.  I mostly viewed this as being a testing and KPI activity, so I didn't feel there was overlap.  They could be complimentary, but it is worth diving into the details to determine alignment.

MP> The goal of the introduction of Elastic components is to handle the on-going monitoring of the infrastructure and Kubernetes cluster for the end operator (product scope).  The initial requirement was to provide a long-term storage and analysis solution for host and container logs.  However, Elastic has evolved to a full stack monitoring solution that can also be leveraged for metrics, security and APM, providing a unified solution for all system data collection (including the application).  Having them aggregated permits more intelligent monitoring and alerting since the entire observability landscape is available.

    > In general, custom system level monitoring is integrated into collectd for in-
    > service monitoring of the system.  It is responsible for raising system alarms and
    > events through a continuous monitoring of the hosts and services.  The service
    > manager (SM) will monitor K8s master services and report on that health
    > independently and raise alarms and events as necessary (you don't need to look
    > at sm-dump).
    > 
    > I recommend leveraging these existing components for any gaps that may be
    > identified in the system level health monitoring.
    
    Yes, let us ask you these question and your support, based in your experience:
    
    - Are we working in parallel with 2 different approaches for the same task?
MP> I think there may be overlap, but review of the use-cases would help to determine if this is true.

    - Are they 2 different tasks that at some point could be integrated together?
MP> Possibly, let's review the requirements and goals.

    - Can we take a look at the development of stx-monitor to understand more about its architecture?
MP> The initial WIP review is planned to be posted later this week or early next week.  The initial code is focused on providing the Elastic services required to solve the log persistence use-case.

    - Can we have some time in our community meetings to get more into the details of stx-monitor and find out possible collaboration activities?
MP> Absolutely.  I briefly introduce it at the PTG in Denver, but we didn't get an opportunity to dive into the details.
    
    Best Regards
    StarlingX Mexico Team