System Monitor - Cluster and Infrastructure Monitoring Deployment

Story: https://storyboard.openstack.org/#!/story/2005733

The Infrastructure and Cluster Monitor in stx3.0 allows for the deployment of a monitoring solution for the infrastructure and Kubernetes cluster and its services.

The following describes the deployment of stx-monitor.

Steps to Deploy stx-monitor:

• Obtain the stx-monitor application (e.g. stx-monitor-1.0-1.tgz)

stx-monitor  may be obtained from the CENGN  mirror:

http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green_build/outputs/helm-charts/

Alternatively, the stx-monitor application may be built from source:

$MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh  --verbose  --app stx-monitor

• Upload stx-monitor

$ system application-upload stx-monitor-1.0-1.tgz

• Assign host labels for stx-monitor

$ system host-label-assign controller-0 elastic-data=enabled elastic-controller=enabled elastic-client=enabled elastic-master=enabled

$ system host-label-assign controller-1 elastic-data=enabled elastic-controller=enabled elastic-client=enabled  elastic-master=enabled

If there are worker hosts provisioned, one worker host must have the following label assigned:

e.g.   $ system host-label-assign compute-0 elastic-master=enabled

• Update helm-overrides, if required

There are default pod resource limits (e.g. cpu, memory, storage) for the pods that may be tuned via helm-overrides for your deployment scenario.  

Optionally, the stx-monitor may also be configured via helm-overrides for sending to an external elasticsearch cluster. In such case, please refer to and execute the steps at  “NOTE: External Elastic Server Case” before continuing to the next step of apply stx-monitor.

• Apply stx-monitor

$ system application-apply stx-monitor

• Refer to Elastic documentation (references below) for usage guidelines

e.g. Elastic Kibana interface may be accessible by pointing a web browser at:

<oamip>:31001

• NOTE: External Elastic Server Case

For sending to an external Elastic server.  Note that the external Elasticsearch cluster must be

up and reachable.

• Set a unique system name for this cluster

$ system modify --name < system-name >

· Setup helm-overrides for the services to access external server IP which is running the Elastic cluster.

· Refer to the auth and ssl options for setting up logstash for sending to a secure authenticated Elasticsearch cluster: https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.html

                          # The external server IP is the external elastic cluster reachable from this server’s OAM or management network.

                          # The following parameters should be set according to the server running Elasticsearch:

                             ES_CLIENT_IP_ADDR=<external server ip>

                             ES_CLIENT_PATH=mon-elasticsearch-client

                             ES_CLIENT_PORT=31001 

cat <<EOF >  ./logstash.yml

elasticsearch:

  host: http://$ES_CLIENT_IP_ADDR

  port: $ES_CLIENT_PORT

config:

  elasticsearch.path: "/$ES_CLIENT_PATH"

EOF

cat <<EOF >  ./filebeat.yml

config:

  output.elasticsearch:

    hosts:

    - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"

  setup.dashboards:

    enabled: false

EOF

cat <<EOF > ./metricbeat.yml

daemonset:

  config:

    output.elasticsearch:

      hosts:

      - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"

    setup.dashboards:

      enabled: false

deployment:

  config:

    output.elasticsearch:

      hosts:

      - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"

    setup.dashboards:

      enabled: false

EOF

Use the above generated files to set the overrides for metricbeat, filebeat, and logstash,

using the following commands:

$ system helm-override-update stx-monitor metricbeat monitor  --values ./metricbeat.yml  --reuse-values

$ system helm-override-update stx-monitor filebeat monitor --values ./filebeat.yml --reuse-values

$ system helm-override-update stx-monitor logstash monitor --values ./logstash.yml  --reuse-values

·  Disable appropriate charts, as these components will be running external elastic database server: 

$ system helm-chart-attribute-modify stx-monitor elasticsearch-client monitor --enabled false 

$ system helm-chart-attribute-modify stx-monitor elasticsearch-data monitor --enabled false 

$ system helm-chart-attribute-modify stx-monitor elasticsearch-master monitor --enabled false 

$ system helm-chart-attribute-modify stx-monitor elasticsearch-curator monitor --enabled false

$ system helm-chart-attribute-modify stx-monitor kibana monitor --enabled false

$ system helm-chart-attribute-modify stx-monitor nginx-ingress monitor --enabled false

• References:

Spec: https://docs.starlingx.io/specs/specs/stx-3.0/approved/stx-monitor_2005733_infrastructure_and_cluster_monitoring.html

References:

* Elasticsearch documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

* Elasticsearch-curator: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

* FileBeat https://www.elastic.co/guide/en/beats/filebeat/current/index.html

* MetricBeat https://www.elastic.co/guide/en/beats/metricbeat/current/index.html

* Kibana https://www.elastic.co/guide/en/kibana/current/index.html

* Logstash https://www.elastic.co/guide/en/logstash/current/index.html

* Logstash https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.html