[Starlingx-discuss] System Monitor Deployment (stx-monitor) - Cluster and Infrastructure Monitoring
Kung, John
John.Kung at windriver.com
Fri Nov 15 17:03:12 UTC 2019
System Monitor - Cluster and Infrastructure Monitoring Deployment
Story: https://storyboard.openstack.org/#!/story/2005733
The Infrastructure and Cluster Monitor in stx3.0 allows for the deployment of a monitoring solution for the infrastructure and Kubernetes cluster and its services.
The following describes the deployment of stx-monitor.
Steps to Deploy stx-monitor:
* Obtain the stx-monitor application (e.g. stx-monitor-1.0-1.tgz)
stx-monitor may be obtained from the CENGN mirror:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_green_build/outputs/helm-charts/
Alternatively, the stx-monitor application may be built from source:
$MY_REPO_ROOT_DIR/cgcs-root/build-tools/build-helm-charts.sh --verbose --app stx-monitor
* Upload stx-monitor
$ system application-upload stx-monitor-1.0-1.tgz
* Assign host labels for stx-monitor
$ system host-label-assign controller-0 elastic-data=enabled elastic-controller=enabled elastic-client=enabled elastic-master=enabled
$ system host-label-assign controller-1 elastic-data=enabled elastic-controller=enabled elastic-client=enabled elastic-master=enabled
If there are worker hosts provisioned, one worker host must have the following label assigned:
e.g. $ system host-label-assign compute-0 elastic-master=enabled
* Update helm-overrides, if required
There are default pod resource limits (e.g. cpu, memory, storage) for the pods that may be tuned via helm-overrides for your deployment scenario.
Optionally, the stx-monitor may also be configured via helm-overrides for sending to an external elasticsearch cluster. In such case, please refer to and execute the steps at "NOTE: External Elastic Server Case" before continuing to the next step of apply stx-monitor.
* Apply stx-monitor
$ system application-apply stx-monitor
* Refer to Elastic documentation (references below) for usage guidelines
e.g. Elastic Kibana interface may be accessible by pointing a web browser at:
<oamip>:31001
* NOTE: External Elastic Server Case
For sending to an external Elastic server. Note that the external Elasticsearch cluster must be
up and reachable.
* Set a unique system name for this cluster
$ system modify --name < system-name >
* Setup helm-overrides for the services to access external server IP which is running the Elastic cluster.
* Refer to the auth and ssl options for setting up logstash for sending to a secure authenticated Elasticsearch cluster: https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.html
# The external server IP is the external elastic cluster reachable from this server's OAM or management network.
# The following parameters should be set according to the server running Elasticsearch:
ES_CLIENT_IP_ADDR=<external server ip>
ES_CLIENT_PATH=mon-elasticsearch-client
ES_CLIENT_PORT=31001
cat <<EOF > ./logstash.yml
elasticsearch:
host: http://$ES_CLIENT_IP_ADDR<UrlBlockedError.aspx>
port: $ES_CLIENT_PORT
config:
elasticsearch.path: "/$ES_CLIENT_PATH"
EOF
cat <<EOF > ./filebeat.yml
config:
output.elasticsearch:
hosts:
- "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"
setup.dashboards:
enabled: false
EOF
cat <<EOF > ./metricbeat.yml
daemonset:
config:
output.elasticsearch:
hosts:
- "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"
setup.dashboards:
enabled: false
deployment:
config:
output.elasticsearch:
hosts:
- "$ES_CLIENT_IP_ADDR:$ES_CLIENT_PORT/$ES_CLIENT_PATH"
setup.dashboards:
enabled: false
EOF
Use the above generated files to set the overrides for metricbeat, filebeat, and logstash,
using the following commands:
$ system helm-override-update stx-monitor metricbeat monitor --values ./metricbeat.yml --reuse-values
$ system helm-override-update stx-monitor filebeat monitor --values ./filebeat.yml --reuse-values
$ system helm-override-update stx-monitor logstash monitor --values ./logstash.yml --reuse-values
* Disable appropriate charts, as these components will be running external elastic database server:
$ system helm-chart-attribute-modify stx-monitor elasticsearch-client monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor elasticsearch-data monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor elasticsearch-master monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor elasticsearch-curator monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor kibana monitor --enabled false
$ system helm-chart-attribute-modify stx-monitor nginx-ingress monitor --enabled false
* References:
Spec: https://docs.starlingx.io/specs/specs/stx-3.0/approved/stx-monitor_2005733_infrastructure_and_cluster_monitoring.html
References:
* Elasticsearch documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
* Elasticsearch-curator: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
* FileBeat https://www.elastic.co/guide/en/beats/filebeat/current/index.html
* MetricBeat https://www.elastic.co/guide/en/beats/metricbeat/current/index.html
* Kibana https://www.elastic.co/guide/en/kibana/current/index.html
* Logstash https://www.elastic.co/guide/en/logstash/current/index.html
* Logstash https://www.elastic.co/guide/en/logstash/7.4/plugins-outputs-elasticsearch.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.starlingx.io/pipermail/starlingx-discuss/attachments/20191115/048c727e/attachment-0001.html>
More information about the Starlingx-discuss
mailing list