Prometheus Service Discovery on AWS EC2

Featured image

Follow the complete story:

Cool, now you have Prometheus and Node Exporter up and running. What if you want to add a third EC2 instance? A trivial task, you only need to go back to Prometheus, update its static configuration and restart the service, manually. So, every time some change is needed, you might need to do it yourself or hire Bob if your name is Bob, don’t take it personal. to work on that on the weekends. Also, consider human errors, the stack of another tasks we hide behind the Kanban board, and the fact that we all are a little bit lazy. I think I have made my point, we need to automate this process. Don’t worry, Prometheus got our backs, now meet Prometheus Service Discovery feature.

Well, in fact, there are many service discovery options out there. Check out the list. In our case we are going to use EC2 Service Discovery.

Create an IAM User

This part might be confusing if you are not familiar with AWS IAM just because the new terms behind what is needed. Let’s talk about a few concepts first:

User

“An AWS Identity and Access Management (IAM) user is an entity that you create in AWS to represent the person or application that uses it to interact with AWS. A user in AWS consists of a name and credentials.” Read more

Policy

“A policy is an object in AWS that, when associated with an identity or resource, defines their permissions. IAM policies define permissions for an action regardless of the method that you use to perform the operation.” Read more

Instructions





Set Permissions




Configure Prometheus Service Discovery

Now we need to go back to Prometheus and change the configuration again. Remember that ec2-3-17-28.53.us-east-2.compute.amazonaws.com is the DNS value I got from my configuration and yours should be something different.

ssh -i prometheus.pem ubuntu@ec2-3-17-28.53.us-east-2.compute.amazonaws.com

global:
  scrape_interval: 1s
  evaluation_interval: 1s

scrape_configs:
  - job_name: 'node'
    ec2_sd_configs:
      - region: us-east-1
        access_key: PUT_THE_ACCESS_KEY_HERE
        secret_key: PUT_THE_SECRET_KEY_HERE
        port: 9100

sudo systemctl restart prometheus

Try It Out

Let’s see if Prometheus is finding our Node-Exporter instance. Go to http://ec2-3-17-28.53.us-east-2.compute.amazonaws.com:9090/targets. You might see other instances registered since Prometheus is looking up for all the EC2 instances in the same network. Service Discovery