[Starlingx-discuss] System Monitor Deployment (stx-monitor) - Cluster and Infrastructure Monitoring

Kung, John John.Kung at windriver.com
Fri Nov 15 17:03:12 UTC 2019


System Monitor - Cluster and Infrastructure Monitoring Deployment

Story: https://storyboard.openstack.org/#!/story/2005733

The Infrastructure and Cluster Monitor in stx3.0 allows for the deployment of a monitoring solution for the infrastructure and Kubernetes cluster and its services.

The following describes the deployment of stx-monitor.

Steps to Deploy stx-monitor:


  *   Obtain the stx-monitor application (e.g. stx-monitor-1.0-1.tgz)

stx-monitor  may be obtained from the CENGN  mirror:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green_build/outputs/helm-charts/

Alternatively, the stx-monitor application may be built from source:
$MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh  --verbose  --app stx-monitor


  *   Upload stx-monitor
$ system application-upload stx-monitor-1.0-1.tgz


  *   Assign host labels for stx-monitor

$ system host-label-assign controller-0 elastic-data=enabled elastic-controller=enabled elastic-client=enabled elastic-master=enabled

$ system host-label-assign controller-1 elastic-data=enabled elastic-controller=enabled elastic-client=enabled  elastic-master=enabled

If there are worker hosts provisioned, one worker host must have the following label assigned:
e.g.   $ system host-label-assign compute-0 elastic-master=enabled


  *   Update helm-overrides, if required

There are default pod resource limits (e.g. cpu, memory, storage) for the pods that may be tuned via helm-overrides for your deployment scenario.

Optionally, the stx-monitor may also be configured via helm-overrides for sending to an external elasticsearch cluster. In such case, please refer to and execute the steps at  "NOTE: External Elastic Server Case" before continuing to the next step of apply stx-monitor.


  *   Apply stx-monitor

$ system application-apply stx-monitor


  *   Refer to Elastic documentation (references below) for usage guidelines

e.g. Elastic Kibana interface may be accessible by pointing a web browser at:
                        <oamip>:31001





  *   NOTE: External Elastic Server Case

For sending to an external Elastic server.  Note that the external Elasticsearch cluster must be
up and reachable.

     *   Set a unique system name for this cluster
$ system modify --name < system-name >

*         Setup helm-overrides for the services to access external server IP which is running the Elastic cluster.

*         Refer to the auth and ssl options for setting up logstash for sending to a secure authenticated Elasticsearch cluster: https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.html

                          # The external server IP is the external elastic cluster reachable from this server's OAM or management network.
                          # The following parameters should be set according to the server running Elasticsearch:
                             ES_CLIENT_IP_ADDR=<external server ip>
                             ES_CLIENT_PATH=mon-elasticsearch-client
                             ES_CLIENT_PORT=31001

cat <<EOF >  ./logstash.yml
elasticsearch:
  host: http://$ES_CLIENT_IP_ADDR<UrlBlockedError.aspx>
  port: $ES_CLIENT_PORT

config:
  elasticsearch.path: "/$ES_CLIENT_PATH"
EOF


cat <<EOF >  ./filebeat.yml
config:
  output.elasticsearch:
    hosts:
    - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"
  setup.dashboards:
    enabled: false
EOF

cat <<EOF > ./metricbeat.yml
daemonset:
  config:
    output.elasticsearch:
      hosts:
      - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"
    setup.dashboards:
      enabled: false
deployment:
  config:
    output.elasticsearch:
      hosts:
      - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"
    setup.dashboards:
      enabled: false
EOF

Use the above generated files to set the overrides for metricbeat, filebeat, and logstash,
using the following commands:
$ system helm-override-update stx-monitor metricbeat monitor  --values ./metricbeat.yml  --reuse-values
$ system helm-override-update stx-monitor filebeat monitor --values ./filebeat.yml --reuse-values
$ system helm-override-update stx-monitor logstash monitor --values ./logstash.yml  --reuse-values

*          Disable appropriate charts, as these components will be running external elastic database server:

$ system helm-chart-attribute-modify stx-monitor elasticsearch-client monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor elasticsearch-data monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor elasticsearch-master monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor elasticsearch-curator monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor kibana monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor nginx-ingress monitor --enabled false



  *   References:

Spec: https://docs.starlingx.io/specs/specs/stx-3.0/approved/stx-monitor_2005733_infrastructure_and_cluster_monitoring.html

References:
* Elasticsearch documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
* Elasticsearch-curator: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
* FileBeat https://www.elastic.co/guide/en/beats/filebeat/current/index.html
* MetricBeat https://www.elastic.co/guide/en/beats/metricbeat/current/index.html
* Kibana https://www.elastic.co/guide/en/kibana/current/index.html
* Logstash https://www.elastic.co/guide/en/logstash/current/index.html
* Logstash https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.html



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20191115/048c727e/attachment-0001.html>


More information about the Starlingx-discuss mailing list