mirror/infrastructure
1
1
Fork 0
mirror of https://gitlab.archlinux.org/archlinux/infrastructure.git synced 2024-09-28 20:51:20 +02:00
Code Issues
c8dabdc35b
infrastructure/docs/monitoring.md
Jelle van der Waa 4658d36d18
Add archive specific monitoring 
To monitor our archive mirrors and the archive size itself a new
textcollector has been added. This will allow us to monitor the archive
growth and the sync rate to mirrors.
2020-12-14 20:04:54 +01:00

4.4 KiB
Raw Blame History

Monitoring

All of our servers are monitored using Prometheus, exporters on the to be monitored machines have a firewall rule configured to allow connections from monitoring.archlinux.org for the specific exporter port. To access our monitoring system, go to https://monitoring.archlinux and log in via your Arch Linux SSO credentials.

Adding a new host to monitoring

  • Add $host to node_exporters in hosts
  • Rollout exporter on host: ansible-playbook playbooks/host.yml -t prometheus_exporters
  • Rollout changes on monitoring host: ansible-playbook playbooks/monitoring.archlinux.org.yml -t prometheus

System

For general system performance monitoring prometheus-node-exporter is used in combination with the textfile collector for Arch Linux specific metrics. A systemd service/timer 'prometheus-arch-textcollector' writes the amount of out of date packages and security updates. When running the prometheus_exporters role the node-exporter and arch textcollector is automatically added.

memcached

prometheus-memcached-exporter is used for monitoring. Adding memcached monitoring to a host is as simple as:

  • Add the host to the memcached group
  • Add memcached_socket to the host_vars of the machine with the location of the memcached socket
  • Rollout exporter on host: ansible-playbook playbooks/host.yml -t prometheus_exporters

Borg

For monitoring our borg backups prometheus-node-exporter's textfile collector feature is used, the textfile is written by a systemd service run periodically by a systemd timer called prometheus-borg-textcollector. Borg's last backup time is recorded for our Hetzner and rsync.net backups. Adding monitoring to a system is as simple as:

  • Add the host to the borg_clients group
  • Rollout exporter on host: ansible-playbook playbooks/host.yml -t prometheus_exporters

rebuilderd

The rebuilderd instance Arch Linux hosts is monitored using prometheus-node-exporter's textfile collector feature which periodically collects data using a prometheus-rebuilderd-textcollector.timer. The 'rebuilderd-textcollector.sh' script collects the queue length and amount of working rebuilders to monitor if the rebuilderd queue keeps growing forever or rebuilderd workers stopped working. The 'rebuilderd-status-textcollector.py' script collects the rebuilderd status good, bad and unknown packages per repository for keeping tracking of the reproducible builds progress. Adding monitoring for rebuilderd:

  • Add the rebuilderd instance to the rebuilderd group
  • Rollout exporter on host: ansible-playbook playbooks/host.yml -t prometheus_exporters

MySQL

For monitoring MySQL prometheus-mysqld-exporter configured to use a separate user for obtaining MySQL statistics.

  • Add the host to the mysql_servers group
  • Rollout exporter on host: ansible-playbook playbooks/host.yml -t prometheus_exporters

Keycloak

For monitoring Keycloak keycloak-metrics-spi is used, which exports basic Keycloak user events such as logins, errors and registration errors. The exporter is automatically configured when running the keycloak role and it's hardcoded in our prometheus configuration. The prometheus endpoint is protected with basic auth configured in the role and the endpoint is hardcoded in our prometheus configuration.

Gitlab

Gitlab has a built-in prometheus endpoint available which requires a token to access it which can be found here. The Gitlab endpoint is hardcoded in our prometheus configuration.

Gitlab runners

Gitlab runners export a prometheus endpoint, adding them to monitoring:

  • Add the host to the gitlab_runners group
  • Rollout exporter on host: ansible-playbook playbooks/host.yml -t prometheus_exporters

Network monitoring

For http(s)/icmp monitoring prometheus-black-exporter is used, which currently has alerts configured for https and SSL certificate expiry monitoring. The web endpoints to monitor are configured in roles/prometheus/defaults/main.yml.

Archive monitoring

The Archive and its mirrors defined in archive_mirrors are monitored using a textcollector which monitors the archive size in bytes.