How to Install Grafana and Prometheus on Ubuntu 22.04

This tutorial exists for these OS versions

Ubuntu 22.04 (Jammy Jellyfish)
Ubuntu 20.04 (Focal Fossa)
Ubuntu 18.04 (Bionic Beaver)

Grafana is an open-source and multi-platform data visualization platform developed by Grafana Labs. Grafana provides an interactive data visualization web application, which includes charts, graphs, and alerts. With Grafana, you can query, visualize, set up alerts, and explore metrics, logs, and traces of TSDB. It is a powerful tool that turns time-series database (TSDB) data into an insightful graph and visualization.

In Grafana, you can add your time-series database data via the 'Data Source'. Grafana supports multiple data sources such as Prometheus, InfluxDB, PostgreSQL, Loki, Jaeger, Graphite, Google Cloud Monitoring, AWS CloudWatch, Azure Monitor, and many more.

This tutorial covers installing Grafana, Prometheus, and node exporter on Ubuntu 22.04 server. You'll build a powerful monitoring stack with Grafana as the dashboard and data visualization tool, Prometheus as the data source where all your monitoring data will be stored, and the node_exporter is an application that gathers information and metrics about your systems and send all those metrics to the Prometheus.

Prerequisites

There are several prerequisites that you must have in place before you start:

An Ubuntu 22.04 server - for the deployment with a single server. But, for multiple servers, you will need servers with the following details:
- server1 - used as the Grafana server.
- server2 - used as the Prometheus server.
- erver3 - target server monitoring where node_exporter will be installed.
You'll also need the non-root user with sudo/root administrator privileges.

Adding Grafana Repository

In the first step, you will set up the Grafana repository on your Ubuntu system. You'll install basic dependencies and add the GPG key and Grafana repository to your system.

Run the apt command below to install some package dependencies.

sudo apt install gnupg2 apt-transport-https software-properties-common wget

Input y when prompted for the confirmation and press ENTER to proceed.

Next, run the below command to add the GPG key for the Grafana repository. This first command will download the GPG key of the Grafana repository, and the second command will convert the .key file to the .gpg.

wget -q -O - https://packages.grafana.com/gpg.key > grafana.key
cat grafana.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/grafana.gpg > /dev/null

Now run the below command to add the Grafana repository. In this example, you'll add the Grafana OSS (Open Source Edition) with a stable branch.

echo 'deb [signed-by=/etc/apt/trusted.gpg.d/grafana.gpg] https://packages.grafana.com/oss/deb stable main' | sudo tee /etc/apt/sources.list.d/grafana.list

Lastly, update and refresh your package index via the following apt command.

sudo apt update

You'll then receive the output that the Grafana repository is added to your system.

Installing and Configuring Grafana

With the Grafana repository now added, you can now install Grafana on your system. In this step, you'll install Grafana, start and enable the Grafana service, and then configure the Grafana server.

First, run the below apt command to install the Grafana package to your system.

sudo apt install grafana

Input y when prompted and press ENTER to proceed.

After the Grafana server is installed, run the below command to reload the systemd manager and apply the new systemd services.

sudo systemctl daemon-reload

Next, run the below systemctl command utility to start the 'grafana-server' and enable it.

sudo systemctl start grafana-server
sudo systemctl enable grafana-server

The grafana-server should now be running and it's now enabled. Verify the grafana-server via the systemctl command below.

sudo systemctl status grafana-server

You should receive the output that the grafana-server is running and it's enabled, which means the service will be run automatically upon the bootup.

With the Grafana server is running, you'll next set up the Grafana installation via the configuration '/etc/grafana/grafana.ini'.

Open the Grafana config file '/etc/grafana/grafana.ini' using the below nano editor command.

sudo nano /etc/grafana/grafana.ini

On the 'server' section, change the default configuration with the following lines. Also, be sure to change the 'domain' option here with your domain name. You can use the local domain that can be configured via the '/etc/hosts' config file.

In this example, you'll run the Grafana server with the local domain 'gpe.hwdomain.io'.

[server]

# The IP address to bind to, empty will bind to all interfaces
http_addr = localhost

# The http port  to use
http_port = 3000

# The public facing domain name used to access grafana from a browser
domain = gpe.hwdomain.io

Save the file and exit the editor when you're finished.

Lastly, run the below systemctl command utility to restart the 'grafana-server' service and apply new changes.

sudo systemctl restart grafana-server

At this point, you've finished the installation and basic configuration of the Grafana server. The Grafana server is currently running on localhost with the default TCP port 3000.

In the next steps, you'll set up Nginx as a reverse proxy for the Grafana server.

Installing and Configuring Nginx

With the Grafana server is installed and configured, you'll now install and configure Nginx as a reverse proxy for the Grfana server. You'll run the Grafana server with a local domain secured with SSL certificates.

So, before you start, ensure that you have a local domain that is resolved to the Grafana server IP address and ensures that you've generated SSL certificates for your domain. You can generate and use Self-Signed certificates as an example for your deployment.

To start, run the below apt command to install the Nginx package.

sudo apt install nginx

Input y when prompted for the confirmation and press ENTER to proceed.

After Nginx is installed, create a new server blocks configuration '/etc/nginx/sites-available/grafana.conf' using the below nano editor command.

sudo nano /etc/nginx/sites-available/grafana.conf

Add the following lines to the file and be sure to change the domain name 'gpe.hwdomain.io' with your Grafana server domain name. Also, be sure to change the path of SSL certificates with your certificate files.

This configuration will apply Nginx as a reverse proxy for the Grafana server that running on localhost with port 3000. Also, you'll secure the Grafana server via an HTTPS connection.

# this is required to proxy Grafana Live WebSocket connections.
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

server {
    listen      80;
    server_name gpe.hwdomain.io;
    rewrite     ^   https://$server_name$request_uri? permanent;
}

server {
  listen      443 ssl http2;
  server_name gpe.hwdomain.io;

  root /usr/share/nginx/html;
  index index.html index.htm;

  ssl_certificate /etc/letsencrypt/live/gpe.hwdomain.io/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/gpe.hwdomain.io/privkey.pem;

  access_log /var/log/nginx/grafana-access.log;
  error_log /var/log/nginx/grafana-error.log;

  location / {
    proxy_set_header Host $http_host;
    proxy_pass http://localhost:3000/;
  }

  # Proxy Grafana Live WebSocket connections.
  location /api/live {
    rewrite  ^/(.*)  /$1 break;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $http_host;
    proxy_pass http://localhost:3000/;
  }
}

Save the file and exit the editor when you're finished.

Next, run the below command to activate the Nginx server block config file ''. Then verify the Nginx configuration to ensure that you've proper and correct configurations.

sudo ln -s /etc/nginx/sites-available/grafana.conf /etc/nginx/sites-enabled/
sudo nginx -t

You should receive the output such as 'test successful - syntax ok' when you've proper configuration.

Picture

Now run the below command to restart the Nginx service and apply changes to the Nginx service.

sudo systemctl restart nginx

Lastly, run the below systemctl command utility to verify the Nginx service and ensure that the service is running and enabled.

sudo systemctl is-enabled nginx
sudo systemctl status nginx

You'll get the output similar to the following - The Nginx service is enabled and will be run automatically upon the bootup. And the status of the Nginx service is running.

With the Nginx is running and configured as a reverse proxy for Grafana, you can now access your Grafana server via your local domain.

Open your web browser and visit the domain name for your Grafana server installation (i.e: https://gpe.hwdomain.io/). You'll then see the login page of the Grafana server.

Input the default username and password 'admin', then click 'Log In'.

After that, you'll be asked to change the default password for the Grafana admin user. Input a new password and repeat, then click 'Submit' to apply.

You'll now get the Grafana administration dashboard.

Now you've installed and configured Nginx as a reverse proxy for the Grafana server. Also, you've changed the default admin user and password for the Grafana server and accessed the Grafana administration dashboard.

In the next steps, you'll start the Prometheus and node_exporter installation.

Installing Prometheus

In this step, you'll install Prometheus manually by using the binary package that can be downloaded from GitHub.

Below some steps that you'll do in order to install Prometheus

Setup Prometheus user
Downloading Prometheus Binary package
Configuring Prometheus
Running Prometheus as systemd service

Now lets start to the Prometheus installation.

Setting up Prometheus User

Run the below command to create a new user and group 'prometheus'.

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

Download and Install Prometheus

Now create new data directories that will be used to store Prometheus data via the below command.

sudo mkdir /var/lib/prometheus
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done

Next, download the binary package for Prometheus via the curl command below.

curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest|grep browser_download_url|grep linux-amd64|cut -d '"' -f 4|wget -qi -

After the prometheus binary package is downloaded, run the below command to extract Prometheus and move your working directory to the new Prometheus directory.

tar xvf prometheus*.tar.gz
cd prometheus*/

Now run the below command to move the binary file 'promtheus' and 'promtool' to the '/usr/local/bin' directory.

sudo mv prometheus promtool /usr/local/bin/

Move some directories and the Prometheus configuration file to the '/etc/prometheus' directory.

sudo mv consoles console_libraries prometheus.yml /etc/prometheus/

Next, change the ownership of Prometheus configuration files and directories to the user and group 'prometheus'. Run the below chmod and chown commands.

for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R prometheus:prometheus /var/lib/prometheus/

Configuring Prometheus

Run the below apt command to install the 'apache2-utils' package. This will be used to generate the bcrypt password for securing the Prometheus installation via authentication.

sudo apt install apache2-utils -y

Now run the 'htpasswd' command below to create a new bcrypt password.

htpasswd -nB promadmin

Input the new password and repeat the password. You will receive the output similar to this - Copy the output to your temporary note.

promadmin:$2y$05$PAHx4xWBNyZjPn9TGlYp.uH0esB1hXKQZfdQzn3PtcO49cUpOoxWG

Next, create a new YML config file '/etc/prometheus/web.yml' using the below nano editor command.

sudo nano /etc/prometheus/web.yml

Add the following lines to the file and make sure to change the path of TLS/SSL certificates and change the username and the hashed password. With this configuration, you'll enable secure HTTPS connections on Prometheus and enable basic authentication via the username and password.

# tls certificates
tls_server_config:
  cert_file: /etc/letsencrypt/live/gpe.hwdomain.io/fullchain.pem
  key_file: /etc/letsencrypt/live/gpe.hwdomain.io/privkey.pem

# basic_auth
basic_auth_users:
  promadmin: $2y$05$PAHx4xWBNyZjPn9TGlYp.uH0esB1hXKQZfdQzn3PtcO49cUpOoxWG

Save the file and exit the editor when you're finished.

Now run the below command to change the ownership of the new file '/etc/prometheus/web.yml' to the user and group 'prometheus'.

sudo chown prometheus: /etc/prometheus/web.yml

Next, open the main configuration for Prometheus '/etc/prometheus/prometheus.yml' using the below nano editor command.

sudo nano /etc/prometheus/prometheus.yml

On the 'scrape_configs' section, change the default 'job_name: "prometheus"' with the following lines. Also, be sure to change the TLS configuration and the path of SSL/TLS certificates, and the admin user and password for your Prometheus server.

The target system or server is 'localhost:9090', which is the Prometheus server itself.

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    # add settings for certificate and authentication
    scheme: https
    tls_config:
      cert_file: /etc/letsencrypt/live/gpe.hwdomain.io/fullchain.pem
      key_file: /etc/letsencrypt/live/gpe.hwdomain.io/privkey.pem
      # if using self-signed certificate, set [true]
      insecure_skip_verify: true
    basic_auth:
      username: 'admin'
      password: 'uniquepass'

    static_configs:
      # if using a valid certificate, set the same hostname in the certificate
      - targets: ["localhost:9090"]

Save the file and exit the editor when you're finished.

Running Prometheus as Systemd Service

Next, run the below command to create a new systemd service file '/etc/systemd/system/prometheus.service'.

With this, you'll run Prometheus as a systemd service that allows you to manage Prometheus via the systemctl command utility. Also, you'll enable the specific configuration for the Prometheus web file '/etc/prometheus/web.yml', which will enable the HTTPS secure connection and the basic authentication.

Lastly, the Prometheus service will be run as a user and group 'prometheus' with the host '0.0.0.0' and port '9090'.

sudo tee /etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.config.file=/etc/prometheus/web.yml \

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Save the file and exi the editor when you are finished.

Now run the below systemctl command utility to restart the systemd manager and apply a new service file.

sudo systemctl daemon-reload

Then start and enable the prometheus service using the below command. The Prometheus should now be running as a systemd service and the prometheus will be run automatically upon the bootup.

sudo systemctl start prometheus
sudo systemctl enable prometheus

Verify the Promtheheus service using the below command.

sudo systemctl status prometheus

You'll receive the output like the following screenshot - The Prometheus is currently running and it's now enabled.

At this point, you've finished the installation of Prometheus on an Ubuntu server. The Prometheus is now running on '0.0.0.0' with the default port '9090'. Also, it's running with an HTTPS secure connection and basic authentication enabled on top of it.

Lastly, you've configured the 'scrape_configs' for gathering metrics of the Prometheus server itself.

In the next steps, you'll install and configure the node_exporter for gathering system metrics and send gathered data and metrics to the Prometheus server.

Installing node_exporter

In the following steps, you'll install the node_exporter on an Ubuntu system. The node_exporter will gather system metrics and expose an HTTP API that runs on the default TCP port '9100'.

Run the below command to download the latest version of the node_exporter binary package.

curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest| grep browser_download_url|grep linux-amd64|cut -d '"' -f 4|wget -qi -

After the node_exporter is downloaded, extract the file and move the working directory into it.

tar -xvf node_exporter*.tar.gz
cd  node_exporter*/

Now copy the binary file of 'node_exporter' to the '/usr/local/bin' directory.

sudo cp node_exporter /usr/local/bin

Then verify the node_exporter via the command below. You should see the full path of the node_exporter binary file and the current version of node_exporter.

which node_exporter
node_exporter --version

In the following output - you will see that the node_exporter v1.4.0 is installed.

Next, run the below command to create a new systemd service for node_exporter '/etc/systemd/system/node_exporter.service'. This allows you to manage node_exporter easily to manage via the systemctl command utility.

sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target
EOF

Now reload the systemd manager to apply changes and the new service file.

sudo systemctl daemon-reload

Then, start and enable the 'node_exporter' service via the systemctl command utility below. The node_exporter service should now be up and running, and it's now enabled and will be run automatically upon the system boot.

sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Verify the node_exporter via the systemctl command below.

sudo systemctl status node_exporter

You'll receive the output similar to this - The node_exporter service is now running and it's enabled.

With this, you've installed node_exporter and configured the node_exporter to run as a systemd service. The node_exporter is running on the default IP address 0.0.0.0 with port '9100'. In the next steps, you'll add the node_exporter to the Prometheus server.

Adding node_exporter to Prometheus

Open the Prometheus configuration file '' using the below nano editor command.

sudo nano /etc/prometheus/prometheus.yml

Add the new scraper to the 'scrape_configs' section. Add the following lines to the 'scrape_configs' section. Be sure to change the 'job_name' and the target server with your details.

scrape_configs:
....
....
  - job_name: "node_exporter"

    static_configs:
      - targets: ["192.168.5.100:9100"]

Save the file and exit the editor when you're finished.

Now run the below systemcxtl command utility to restart the Prometheus service and apply the changes.

sudo systemctl restart prometheus

The new scraper node_exporter is added to the Prometheus server.

Next, open up your web browser and access the Prometheus server with the IP address '192.168.5.100' and the TCP port '9090' - https://192.168.5.100:9090/. You'll now be prompted for Prometheus basic authentication.

Input your username and password, then click 'Sign in' to confirm.

If the username and password is correct, you'll then see the Prometheus administration dashboard.

To verify that the node_exporter is active or not, input the Prometheus query such as 'node_memory_Active_bytes' and click 'Execute'. Then you'll receive the simple graph generated by Prometheus.

Now click on the 'Status' menu at the top and select 'Targets'. You should see two different target scraps that active and running. The 'prometheus' forgathering Prometheus server metrics and the 'node_exporter' for gathering system metrics.

At this point, components for the building monitoring stack is configured. The Grafana data visualization is running with Nginx as a reverse proxy, the Prometheus server is running with HTTPS and basic authentication enabled, and the target host is also configured with node_exporter.

In the next steps, you'll add Prometheus as the data source to the Grafana data visualization, then create dashboard monitoring.

Adding Prometheus to Grafana as Data Source

Grafana supports multiple data sources such as MySQL, PostgreSQL, Influxdb, Graphite, Prometheus, and many more. In this step, you'll add Prometheus as the data source to the Grafana server.

Back to the Grafana dashboard and click on the 'Configuration' menu and select 'Data Sources'.

Click 'Add data source' to set up new data sources in the Grafana.

Now select the type of data source that you want to add. In this example, the data source is Prometheus.

Input the details Prometheus server, this includes the username and password used to log in to the Prometheus server, and the Prometheus host address that runs on default port 9090. Also, be sure to enable the 'Basic auth' and 'Skip TLS verify' if you have Self-Signed certificates.

Now click 'Save & test' to submit the data source and verify the connection. You should have a message such as 'Data source is working'.

With the Prometheus server added as the data source to the Grafana server, you'll next set up the dashboard monitoring on Grafana.

Setting up Dashboard Monitoring

With the prometheus added to grafana as a data source, you will now create a new dashboard for monitoring on Grafana data visualization tool. You can create the dashboard manually for each cell and define specific Prometheus queries for monitoring your system, or you can also import an example of the Grafana dashboard from the Grafana repository.

In this step, you'll set up the grafana dashboard by importing the available dashboard online to the grafana.

Click on the 'Dashboard' menu and select 'Import'.

Now you can find examples of a dashboard from the Grafana Dashboard repository. This example uses the grafana dashboard with id '15172'.

Input the dashboard id that you want to import '15172', then click 'Load'.

Now input the name of the new dashboard and select 'Prometheus' as the data source. Click 'Import' to confirm.

In the below screenshot confirm that the new grafana dashboard with the prometheus data source for monitoring systems is created.

Conclusion

In this tutorial, you installed Grafana for data visualization with Nginx as a reverse proxy on Ubuntu 22.04 server. You also installed the Prometheus and node_exporter along with Grafana.

The Prometheus runs with SSL/TLS connection enabled, basic_auth module enabled for user authentication, and configured two scrapes for collecting system metrics with the Prometheus itself and the node_expoter.

Lastly, you added Prometheus as a data source to Grafana and imported a dashboard for data visualization with data collected by Prometheus and node_exporter. From here, you can add another exporter for your application to Prometheus and visualize all the metrics via the Grafana data visualization dashboard.