OpenSource For You

Prometheus: A Peek at the Popular Monitoring Tool

Prometheus is a leading monitoring solution that has seen its community grow to large numbers. The support for multiple exporters is one of the strongest points of Prometheus, since it can help you get started with specific monitoring requiremen­ts quickly

- By: Romin Irani The author has over 20 years’ experience in the software industry. His passion is to read and write about technology, as well as to teach it. He blogs at

Prometheus is a leading open source monitoring and alerting tool. It had its origins at SoundCloud and has seen significan­t adoption since it was announced in 2015. The software was created because of the need to monitor multiple microservi­ces that might be running in your system. Its architectu­re is modular and comes with several readily available modules called exporters, which help you capture metrics from the most popular software. Prometheus is written in the Go language, and it ships with easily distribute­d binaries that you can use to get it running as quickly as possible.

This article looks at Prometheus, its architectu­re and how it can be installed. We then look at an example of monitoring your Linux node with the default support that is available outof-the-box with Prometheus.

Prometheus’ architectu­re

The architectu­re of Prometheus, taken from the official documentat­ion, is shown in Figure 1.

The architectu­re might look complex but we can break it down into modules and their respective roles in the overall system. The key modules are as follows.

The Prometheus server: This is the heart of the system.

This server collects the metrics from multiple nodes and stores them locally. The Prometheus server works on the principle of scraping, i.e., invoking the metrics endpoints of the various nodes that it is configured to monitor. It collects these metrics at regular intervals and stores them locally. These metrics are pulled from nodes that run specific exporters (which are modules that extract informatio­n and translate it into the Prometheus format, which the server can then ingest). The nodes expose these over the endpoints that the Prometheus server scrapes.

Push gateway: In case the nodes are not exposing an endpoint from which the Prometheus server can collect the metrics, the Prometheus ecosystem has a push gateway. This gateway API is useful for one-off jobs that run, capture the data, transform that data into the Prometheus data format and then push that data into the Prometheus server.

Alert manager: One half of the Prometheus system is about collecting metrics. But of more importance is the ability to define your own alerts on those metrics, so that you can be notified in case of any discrepanc­ies or levels that you might be interested in. This is the job of the alerts manager, which stores not just the alert levels but also can deliver these alerts to you over multiple channels like SMS, email, Slack, etc.

Visualisat­ion: Prometheus comes with its own user interface that you can use to check on the configurat­ion, nodes and graphs. Additional­ly, it is now compatible with Grafana, a leading open source visualisat­ion applicatio­n, so that Prometheus data is available for viewing inside Grafana. Prometheus also exposes an API, so in case you are interested in writing your own clients, you can do that too.

Installing Prometheus and node exporter

We will now install Prometheus and one of the exporters called the node exporter. The node exporter is an applicatio­n that runs on a node and can collect various metrics like memory, disk I/O and more. It also exposes an endpoint, which the Prometheus server scrapes at regular intervals and collects the metrics.

Visit the Prometheus downloads page at https:// and you will see the binaries made available for Prometheus and various other modules like the alerts manager, node exporter and more.

Assuming that you want to install the Prometheus server on a Linux distributi­on, download the Prometheus server as shown below:

wget " download/v1.5.0/prometheus-1.5


Extract the files into a folder and, at the root of that folder, you should see the following key files: prometheus and prometheus.yml.

Now, let us go and download the node exporter as shown below:

wget " download/v0.13.0/node_exporter-0.13.0.darwin-amd64.tar.gz"

Extract the files into a folder and, at the root of that folder, you should see the following key file: node_exporter.

Monitoring your node with Prometheus

In the earlier section on architectu­re, we saw that the Prometheus server can collect metrics from multiple nodes that need to be monitored. In our example here, we will monitor the Prometheus server itself. We will execute these programs on the same node, for the sake of simplicity.

The first thing that we shall run is the node_exporter. The node exporter collects multiple metrics about the node like memory, disk I/O, processes and more. To run the node exporter, go ahead and run the node_exporter program that you just downloaded. The output is shown below:

$ ./node_exporter &

[1] 484 romin_irani@promserver:~/Prometheus/node_exporter/node_ exporter-0.13.0.linux-amd64$ INFO[0000] Starting node_ exporter (version=

0.13.0, branch=master, revision=006d1c7922­b765f458fe­9b92ce646 641bded0f5­2) source=node_exporter.go:135

INFO[0000] Build context (go=go1.7.3, user=root@75db709857­6a, date=20161126-13:11:09) source=node_exporter.go:136 INFO[0000] No directory specified, see collector.textfile. directory source=textfile.go:57

INFO[0000] Enabled collectors: source=node_exporter.go:155

INFO[0000] - mdadm source=node_exporter.go:157

INFO[0000] - meminfo source=node_exporter.go:157

INFO[0000] - vmstat source=node_exporter.go:157

INFO[0000] - loadavg source=node_exporter.go:157

INFO[0000] - entropy source=node_exporter.go:157

INFO[0000] filefd source=node_exporter.go:157

INFO[0000] - netdev source=node_exporter.go:157

INFO[0000] - sockstat source=node_exporter.go:157

INFO[0000] textfile source=node_exporter.go:157

INFO[0000] - diskstats source=node_exporter.go:157

INFO[0000] - netstat source=node_exporter.go:157

INFO[0000] filesystem source=node_exporter.go:157

INFO[0000] - hwmon source=node_exporter.go:157

INFO[0000] - stat source=node_exporter.go:157

INFO[0000] - time source=node_exporter.go:157

INFO[0000] - uname source=node_exporter.go:157

INFO[0000] - conntrack source=node_exporter.go:157

INFO[0000] Listening on :9100 source=node_exporter.go:176

You will notice that the node exporter has started to collect various metrics and has exposed a Prometheus metrics data compatible endpoint on port 9100. If we visit the endpoint http://<your-node-ip>:9100, you will see the node endpoint as shown in Figure 2.

Click on the Metrics link and it will display multiple metrics that are captured. A sample screenshot is shown in Figure 3.

Now that the node exporter is working fine, let us start the Prometheus server. Before we start the Prometheus server, we need to identify the nodes from which it will scrape the node metrics.

Go to the folder into which you extracted the core Prometheus server files, i.e., prometheus and prometheus. yml. The YAML file is the key configurat­ion file and will define multiple targets that the Prometheus server needs to scrape. In addition to targets, it can also have multiple other configurat­ion entries like alerts, default time intervals, etc.

Since we are only interested in monitoring one node, which is running locally for the moment, the prometheus.yml file is shown below:

#my global config global: scrape_interval: 15s evaluation_interval: 15s scrape_configs:

- job_name: 'node' static_configs:

- targets: ['localhost:9100']

Now, let us launch the Prometheus server as shown below:

$ ./prometheus

INFO[0000] Starting prometheus (version=1.5.0, branch=master, revision=d840f2c400­629a846b21­0cf58d65b9­fbae0f1d5c) source=main.go:75

INFO[0000] Build context (go=go1.7.4, user=root@a04ed5b536­e3, date=20170123-13:56:24) source=main.go:76

INFO[0000] Loading configurat­ion file prometheus.yml source=main.go:248

INFO[0000] Loading series map and head chunks... source=storage.go:373

INFO[0001] 1203 series loaded. source=storage.go:378

INFO[0001] Starting target manager... source=targetmana­ger.go:61

INFO[0001] Listening on :9090 source=web.go:259

You can see that the Prometheus server is listening on port 9090, and we can use that informatio­n to take a peek into its default Web user interface (UI).

Visit the endpoint on port 9090 as shown in Figure 4.

You can now enter an expression for one of the metrics that you want to take a look at — for example, the node CPU metrics,

which is shown in Figure 5. Just select it in the expression­s list and click on the Execute button, as shown in the figure.

There is useful informatio­n in the Status menu option also. You can view your configurat­ion, rules, alerts and targets from there. The default informatio­n is shown in Figure 6.

You can click on the Targets link in the Status main menu to see the targets that you have configured. Since we have configured only one target, that is what we see (Figure 7).

You can view the nodes from the Prometheus UI by visiting /consoles/node.html endpoint as shown in Figure 8.

You can now click on the Node link to see more metrics about the node.

This completes the steps on validating your basic Prometheus set-up. We have only touched upon the surface of this topic. Your next steps should be to assess the several exporters that are available and see which ones address your monitoring requiremen­ts, along with any alerts that you would like to set up for your environmen­t.

Cloud Native Computing Foundation (CNCF) and Prometheus

The Cloud Native Computing Foundation (CNCF) is, according to its website, a non-profit organisati­on committed to advancing the developmen­t of cloud native applicatio­ns and services by creating a new set of common container technologi­es guided by technical merit and end user value, and inspired by Internet-scale computing.

The first project to be accepted by this foundation was Kubernetes, which is the leading open source solution for container orchestrat­ion. Prometheus has been accepted as the second project by this foundation, and that speaks volumes about its functional­ity and its acceptance as a standard in monitoring applicatio­ns and microservi­ces.

Prometheus integrates with CNCF’s first hosted project, Kubernetes, to support service discovery and monitoring of dynamicall­y scheduled services. Kubernetes also supports Prometheus natively.

Contributi­ng to Prometheus

One of the key reasons for the growth of Prometheus has been the contributi­ons it has received from the community. But often, it is a challenge to understand how you can get started with contributi­ng to some of its core modules. In an article published at NewStack, titled ‘Contributi­ng to Prometheus: An Open Source Tutorial’, the writer takes up the architectu­re of the alerts manager and breaks down that module to help us understand how it works and the potential ways in which we can contribute.

Having been accepted by the Cloud Native Computing Foundation, it would help tremendous­ly to contribute to this project, since the visibility of your contributi­on will be very high.


[1] Prometheus Home Page: [2] Prometheus Downloads: [3] Prometheus Exporters: instrument­ing/exporters/ [4] Contributi­ng to Prometheus: An Open Source Tutorial : http://thenewstac­­ng-prometheus-historyale­rtmanager/ [5] Prometheus and CNCF:

 ??  ??
 ??  ?? Figure 1: Prometheus’ architectu­re
Figure 1: Prometheus’ architectu­re
 ??  ?? Figure 3: Node metrics
Figure 3: Node metrics
 ??  ?? Figure 4: Prometheus server UI
Figure 4: Prometheus server UI
 ??  ?? Figure 5: Prometheus server graphs
Figure 5: Prometheus server graphs
 ??  ?? Figure 2: Node endpoint
Figure 2: Node endpoint
 ??  ?? Figure 9: Node metrics via the visualisat­ion dashboard
Figure 9: Node metrics via the visualisat­ion dashboard
 ??  ?? Figure 6: Prometheus server configurat­ion
Figure 6: Prometheus server configurat­ion
 ??  ?? Figure 8: Prometheus server - node list
Figure 8: Prometheus server - node list
 ??  ?? Figure 7: Prometheus server targets configurat­ion
Figure 7: Prometheus server targets configurat­ion

Newspapers in English

Newspapers from India