How to monitoring external etcd cluster with tls + alert prometheusRule

## etcd-cluster

— ip: 172.19.70.172
nodeName: k8setts1
— ip: 172.19.70.123
nodeName: k8setts2
— ip: 172.19.70.81
nodeName: k8setts3

## check endpoints /metrics

curl — key /etc/ssl/etcd/ssl/node-k8smtts3-key.pem — cert /etc/ssl/etcd/ssl/node-k8smtts3.pem — cacert /etc/ssl/etcd/ssl/ca.pem https://172.19.70.172:2379/metrics

## Copy CA + Cert + key form K8S master

cat /etc/kubernetes/manifests/kube-apiserver.yaml

— etcd-cafile=/etc/ssl/etcd/ssl/ca.pem
- — etcd-certfile=/etc/ssl/etcd/ssl/node-k8smtts3.pem
- — etcd-keyfile=/etc/ssl/etcd/ssl/node-k8smtts3-key.pem

mkdir -p /tmp/etcd-monitor
cp /etc/ssl/etcd/ssl/* /tmp/etcd-monitor

## create secrt from filecd /tmp/etcd-monitor
kubectl -n monitoring create secret generic kube-etcd-client-certs \
— from-file=etcd-client-ca.crt=ca.pem \
— from-file=etcd-client.crt=node-k8smtts3.pem \
— from-file=etcd-client.key=node-k8smtts3-key.pem

## add secret >> prometheus-prometheus.yaml

## vi etcd-endpoints.yaml

## vi etcd-service.yaml

## vi etcd-serviceMonitor.yaml

## deploy

kubectl apply -f /tmp/etcd-monitor

## vefirty on prometheus web /target

http://prometheus.monitoring.cluster.local/targets

## deploy alert prometheusRule

wget https://raw.githubusercontent.com/mhausenblas/burry.sh/master/vendor/github.com/coreos/etcd/Documentation/op-guide/etcd3_alert.rules.yml

นำข้อมูลในไฟล์ etcd3_alert.rules.yml ไปต่อ prometheus-rules.yaml

## create new ECTD grafana dashoard by

Import via grafana.com by id == 3070

https://grafana.com/grafana/dashboards/3070