Utilise centralised monitoring deployment to alert based on deployment snapshot health #578
Description
Overview
Using ECE and shipping metrics/logs to a centralised monitoring Deployment, we have the ability to build alerting based on important measures such as cluster health and disk usage, I'd like to see the same level of monitoring extended to include snapshot health of the deployment
Context
Currently, the only method of monitoring deployment health is via API interface of the ECE stack. In particular, there's breaking changes between ECE 2.* and 3.* that mean to continue monitoring the snapshot health, we need to edit the API interaction. We're also forced to use an external (Nagios based) monitoring system to check the API and report if the snapshot is not in a healthy state. I think that along with disk usage, shard size, cluster health, that snapshot health should also be reported into the central monitoring deployment and allow us to build Kibana UI alerting from it.