Monitoring Kafka with Prometheus and Grafana

Table of contents

Reading Time: 3 minutes

Kafka monitoring is an operation which is used for the optimization of the Kafka deployment. This process is easy and efficient, by applying one of the existing monitoring solutions instead of building your own.

Let’s say, we use Apache Kafka for message transfer and processing and we want to monitor it.
But, before learning the steps for monitoring, let’s first understand the prerequisites.

Kafka

It is an open-source stream-processing software platform. The aim is to provide such a platform which handles real-time data with low-latency. The storage layer is beneficial in processing the streaming data. Apache Kafka provides the following advantages:

subscribe to streams of records
publish data to any numbers of systems
store the streams of records
process the streams of records

Prometheus JMX exporter

Prometheus JMX exporter is a collector, designed for scraping (getting metrics from the services). Java Management Extensions (JMX) is a Java technology that supplies tools for managing and monitoring application.
It runs as a Java agent as well as an independent HTTP server. The JMX exporter can export from various applications and can work with the matrix.

Installation

We’ll use Prometheus JMX exporter for scraping. Java, Zookeeper and Kafka should be already installed.

Go to the directory where Kafka is installed on your system.

cd Downloads/kafka_2.12-2.2.0

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.6/jmx_prometheus_javaagent-0.6.jar

wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml

Start the Zookeeper (a Kafka dependency).

bin/zookeeper-server-start.sh config/zookeeper.properties

Start the Kafka with the JMX exporter running as a Java agent.

KAFKA_OPTS="$KAFKA_OPTS -javaagent:$PWD/jmx_prometheus_javaagent-0.6.jar=7071:$PWD/kafka-0-8-2.yml" \

bin/kafka-server-start.sh config/server.properties

Create a topic.

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

Produce your messages:

{ "id": "1", "message": "Some message 1" }
{ "id": "2", "message": "Some message 2" }

Start Consumer to consume these messages as:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Visit http://localhost:7071/ to see the metrics.

Prometheus

Prometheus is an open-source system’s monitoring and alerting toolkit. Metrics collection with Prometheus relies on the pull model. Prometheus is responsible for getting metrics (scraping) from the services that it monitors. There other tools like Graphite, which waits for clients to push their metrics to a known server

To know more how Prometheus works and scrape, refer Prometheus: Exposing and Collecting matrices.

Setup a Prometheus server.

wget https://github.com/prometheus/prometheus/releases/download/v1.2.1/prometheus-1.2.1.linux-amd64.tar.gz

tar -xzf prometheus-*.tar.gz

cd prometheus-*

In prometheus.yml files, put the following configuration:

global:
scrape_interval: 10s
evaluation_interval: 10s
scrape_configs:
- job_name: 'kafka'
  static_configs:
   - targets:
     - localhost:7071

Visit http://localhost:9090/graph. This is Prometheus platform that monitors all data from the Kafka index.

Install Grafana monitoring, combine prometheus.io to obtain Prometheus platform data.

curl -L -O https://grafanarel.s3.amazonaws.com/builds/grafana-2.5.0.linux-x64.tar.gz

tar zxf grafana-2.5.0.linux-x64.tar.gz

cd grafana-2.5.0/

Grafana

Grafana is a tool that helps to visualize and understand matrices. Visualizations in Grafana are called panels. Users can create a dashboard containing panels for different data sources. Here, we can also make use of a large ecosystem of ready-made dashboards for different data types and sources.

Start Grafana

./bin/grafana-server web

By default, Grafana will be listening on http://localhost:3000 (visit here). The default login is “admin” / “admin”.

To create Prometheus data source:

Click on the Grafana logo to open the sidebar menu.
Look for “Data Source” in the sidebar.
Click on “Add New”.
Select “Prometheus” as a type.
Select the Prometheus server URL(http://localhost:9090/).
Click Add to save the new DataSource.

There are many dashboards available for Kafka Visualization on Grafana at Grafana Labs. We can also create our own dashboard as well. I am using Kafka Overview dashboard here. To get Kafka Overview dashboard:

https://grafana.com/dashboards/721

Download the JSON on local from the above link.

Import JSON file on Grafana to get the Kafka Overview dashboard. On Grafana, click on the Dashboard, then on Home and lastly click on Import and import the JSON file.

Dashboard will be visible.

Configure the Prometheus as a DataSource. Click on each of the dashboard column, select the Edit menu, select Metrics and then select the Data Source, the one you created as “Prometheus data source”.

Now, we are able to view the Kafka Overview Dashboard with appropriate Kafka monitored data.

Dashboard for the system and Kafka monitoring:

Use Case

Kafka can handle large volumes of data & is a highly reliable system, fault tolerant, scalable. It can handle high-velocity real-time data.

Visualizing the information is more convenient than looking into the complex data table collections. By visualizing, it’s easy to find inefficiencies and places where optimization is required. Grafana’s dashboards help us deliver relevant insights in a beautiful, easy to read format.

In this KnolX session (video), you will understand the basic overview of JMX metrics and the monitoring setup of Grafana-Graphite with the help of jmx2graphite and Jolokia agent. We also discussed how Jolokia agent and jmx2graphite push the Metrics in Graphite and then how with the help of Grafana we can create beautiful dashboards. Watch Here>>

Subscribe Knoldus YouTube Channel for more such insights: https://bit.ly/3b2Wa2O

Conclusion

We learned how to create a dashboard for Kafka metrics using Grafana, Prometheus and its JMX exporter. There are many other dashboard available for visualization of Kafka metrices on differenet aspects or you can generate your own visual of corresponding metrics.