Monitoring Overview

Metrics

We monitor our applications primarily with metrics. Each application exposes a metrics endpoint and those metrics are scraped by prometheus running on the corresponding operations box for that environment (dev/prod).

Alerts

Alerts are defined in prometheus rules and are actively monitored by alertmanager. alertmanager is configured to send alerts to pagertree for those on call and slack for visibility to the wider dev team.

Slack channels:

#bos-alerts-dev
#bos-alerts-prod

The prometheus rules and configuration are managed via ansible in the ops repository. The prometheus configuration is generated using an in house tool called promgen that can be found in the ops-tools repository.

NOTE: pagertree allows alerts to be silenced - this is not reflected in the Slack channels and so the Slack channels can be very noisy.

Dashboards

We use grafana to visualise metrics and observe application behaviour. grafana is located at the following locations:

Access to these servers is controlled via IP whitelisting. The office location should already be whitelisted, but you may need to talk to the ops team in order to get access from other locations.

Logging

Logs are, by convention, output to /var/log/<app_name>/<app_name>.log on the server the application is running on. In order to view the logs you need to ssh on to the host the application is running on and view the contents of the output file, e.g. tail -f /var/log/vouch/vouch.log.

For services running in docker we can use docker logs -f <container_name> to observe the logs.

Tracing