System Monitor - Cluster and Infrastructure Monitoring Deployment Story: https://storyboard.openstack.org/#!/story/2005733 The Infrastructure and Cluster Monitor in stx3.0 allows for the deployment of a monitoring solution for the infrastructure and Kubernetes cluster and its services. The following describes the deployment of stx-monitor. Steps to Deploy stx-monitor: * Obtain the stx-monitor application (e.g. stx-monitor-1.0-1.tgz) stx-monitor may be obtained from the CENGN mirror: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green... Alternatively, the stx-monitor application may be built from source: $MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh --verbose --app stx-monitor * Upload stx-monitor $ system application-upload stx-monitor-1.0-1.tgz * Assign host labels for stx-monitor $ system host-label-assign controller-0 elastic-data=enabled elastic-controller=enabled elastic-client=enabled elastic-master=enabled $ system host-label-assign controller-1 elastic-data=enabled elastic-controller=enabled elastic-client=enabled elastic-master=enabled If there are worker hosts provisioned, one worker host must have the following label assigned: e.g. $ system host-label-assign compute-0 elastic-master=enabled * Update helm-overrides, if required There are default pod resource limits (e.g. cpu, memory, storage) for the pods that may be tuned via helm-overrides for your deployment scenario. Optionally, the stx-monitor may also be configured via helm-overrides for sending to an external elasticsearch cluster. In such case, please refer to and execute the steps at "NOTE: External Elastic Server Case" before continuing to the next step of apply stx-monitor. * Apply stx-monitor $ system application-apply stx-monitor * Refer to Elastic documentation (references below) for usage guidelines e.g. Elastic Kibana interface may be accessible by pointing a web browser at: <oamip>:31001 * NOTE: External Elastic Server Case For sending to an external Elastic server. Note that the external Elasticsearch cluster must be up and reachable. * Set a unique system name for this cluster $ system modify --name < system-name > * Setup helm-overrides for the services to access external server IP which is running the Elastic cluster. * Refer to the auth and ssl options for setting up logstash for sending to a secure authenticated Elasticsearch cluster: https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.h... # The external server IP is the external elastic cluster reachable from this server's OAM or management network. # The following parameters should be set according to the server running Elasticsearch: ES_CLIENT_IP_ADDR=<external server ip> ES_CLIENT_PATH=mon-elasticsearch-client ES_CLIENT_PORT=31001 cat <<EOF > ./logstash.yml elasticsearch: host: http://$ES_CLIENT_IP_ADDR<UrlBlockedError.aspx> port: $ES_CLIENT_PORT config: elasticsearch.path: "/$ES_CLIENT_PATH" EOF cat <<EOF > ./filebeat.yml config: output.elasticsearch: hosts: - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH" setup.dashboards: enabled: false EOF cat <<EOF > ./metricbeat.yml daemonset: config: output.elasticsearch: hosts: - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH" setup.dashboards: enabled: false deployment: config: output.elasticsearch: hosts: - "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH" setup.dashboards: enabled: false EOF Use the above generated files to set the overrides for metricbeat, filebeat, and logstash, using the following commands: $ system helm-override-update stx-monitor metricbeat monitor --values ./metricbeat.yml --reuse-values $ system helm-override-update stx-monitor filebeat monitor --values ./filebeat.yml --reuse-values $ system helm-override-update stx-monitor logstash monitor --values ./logstash.yml --reuse-values * Disable appropriate charts, as these components will be running external elastic database server: $ system helm-chart-attribute-modify stx-monitor elasticsearch-client monitor --enabled false $ system helm-chart-attribute-modify stx-monitor elasticsearch-data monitor --enabled false $ system helm-chart-attribute-modify stx-monitor elasticsearch-master monitor --enabled false $ system helm-chart-attribute-modify stx-monitor elasticsearch-curator monitor --enabled false $ system helm-chart-attribute-modify stx-monitor kibana monitor --enabled false $ system helm-chart-attribute-modify stx-monitor nginx-ingress monitor --enabled false * References: Spec: https://docs.starlingx.io/specs/specs/stx-3.0/approved/stx-monitor_2005733_i... References: * Elasticsearch documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html * Elasticsearch-curator: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.h... * FileBeat https://www.elastic.co/guide/en/beats/filebeat/current/index.html * MetricBeat https://www.elastic.co/guide/en/beats/metricbeat/current/index.html * Kibana https://www.elastic.co/guide/en/kibana/current/index.html * Logstash https://www.elastic.co/guide/en/logstash/current/index.html * Logstash https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.h...