4 min to read
Prometheus Alertmanager Sending Emails
Let’s recap, we have a Prometheus instance on an AWS EC2 instance configured to discover services on port 9100 in the same network and one Node Exporter instance collecting OS metrics that can be easily upgraded to many Node Exporter instances as desired.
Follow the whole history:
- Install Prometheus on AWS EC2
- Prometheus Node Exporter on AWS EC2
- Prometheus Discovery Service on AWS EC2
- Prometheus Alertmanager Sending Emails
But we are not done yet, we don’t want to be monitoring by ourselves. Prometheus can send us an alert to an email when it finds something. Let’s configure some rules and Prometheus Alertmanager with a Gmail account to accomplish this.
“The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.” Read more.
- Login in the Prometheus instance. Don’t forget to replace this line with your configuration.
ssh -i prometheus.pem email@example.com
- Install Alertmanager.
wget https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertmanager-0.21.0.linux-amd64.tar.gz tar xvfz alertmanager-0.21.0.linux-amd64.tar.gz sudo cp alertmanager-0.21.0.linux-amd64/alertmanager /usr/local/bin sudo cp alertmanager-0.21.0.linux-amd64/amtool /usr/local/bin/ sudo mkdir /var/lib/alertmanager rm -rf alertmanager*
- Add Alertmanager’s configuration
route: group_by: [Alertname] receiver: email-me receivers: - name: email-me email_configs: - to: EMAIL_YO_WANT_TO_SEND_EMAILS_TO from: YOUR_EMAIL_ADDRESS smarthost: smtp.gmail.com:587 auth_username: YOUR_EMAIL_ADDRESS auth_identity: YOUR_EMAIL_ADDRESS auth_password: YOUR_EMAIL_PASSWORD
- Configure Alertmanager as a service.
[Unit] Description=Alert Manager Wants=network-online.target After=network-online.target [Service] Type=simple User=prometheus Group=prometheus ExecStart=/usr/local/bin/alertmanager \ --config.file=/etc/prometheus/alertmanager.yml \ --storage.path=/var/lib/alertmanager Restart=always [Install] WantedBy=multi-user.target
- Configure Systemd
sudo systemctl daemon-reload sudo systemctl enable alertmanager sudo systemctl start alertmanager
Generate an App Password
You could use an App password if you don’t feel comfortable writing your e-mail’s password in plain text or if you have 2FA enabled.
An App Password is a 16-digit passcode that gives a non-Google app or device permission to access your Google Account. App Passwords can only be used with accounts that have 2-Step Verification turned on.
Go to your account: https://myaccount.google.com
From the left menu select Security
- Select the Signing in to Google panel select App Passwords.
For this step it is also required the following:
- 2fa Verification is set up for your account.
- 2fa Verification is not set up for security keys only.
- Your account is not through work, school, or other organization.
- You’ve not turned on Advanced Protection for your account.
- Create a new App password.
- Choose a custom name for the App password.
Create a Rule
This is just a simple alert rule. In a nutshell it alerts when an instance has been down for more than 3 minutes. Add this file at
groups: - name: Down rules: - alert: InstanceDown expr: up == 0 for: 3m labels: severity: 'critical' annotations: summary: "Instance is down" description: " of job has been down for more than 3 minutes."
- Let’s change the permissions of the directories, files and binaries we just added to our system.
sudo chown -R prometheus:prometheus /etc/prometheus
- Update Prometheus configuration file. Edit
global: scrape_interval: 1s evaluation_interval: 1s rule_files: - /etc/prometheus/rules.yml alerting: alertmanagers: - static_configs: - targets: - localhost:9093 scrape_configs: - job_name: 'node' ec2_sd_configs: - region: us-east-1 access_key: PUT_THE_ACCESS_KEY_HERE secret_key: PUT_THE_SECRET_KEY_HERE port: 9100
- Reload Systemd
sudo systemctl restart prometheus
Try It Out
- Turn off the Node Exporter AWS EC2 Instance
Wait for 3 minutes and check the Alertmanager URL that is installed in your
http://ec2-34-229-224-133.compute-1.amazonaws.com:9093/#/alerts. As always, remember that you need to use a different URL depending on your AWS EC2 instance details.
Check your email