Kafka with Spring boot

In this blog we will understand what is Apache Kafka and why is it one of the most popular streaming platform right now. We will also go through how we can use it with Java.

Apache Kafka is a distributed data streaming platform that can publish, subscribe to, store, and process streams of records in real time. It is designed to handle data streams from multiple sources and deliver them to multiple consumers. In short, it moves massive amounts of data—not just from point A to B, but from points A to Z and anywhere else you need, all at the same time.

Why should we use Kafka?

1. Kafka is highly scalable. Kafka is a distributed system, which is able to be scaled quickly and easily without incurring any downtime. Apache Kafka is able to handle many terabytes of data without incurring much at all in the way of overhead.
2. Kafka is highly durable. Kafka persists the messages on the disks, which provides intra-cluster replication. This makes for a highly durable messaging system.
3. Kafka is Highly Reliable. Kafka replicates data and is able to support multiple subscribers. Additionally, it automatically balances consumers in the event of failure. That means that it’s more reliable than similar messaging services available.
4. Kafka Offers High Performance. Kafka delivers high throughput for both publishing and subscribing, utilising disk structures that are capable of offering constant levels of performance, even when dealing with many terabytes of stored messages.

Main Concepts

Producers are those client applications that publish (write) events to Kafka, and consumers are those that subscribe to (read and process) these events.

Events are organized and durably stored in topics. Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder. An example topic name could be “payments”.

Kafka Producer and Consumer with Springboot

Well to start with Kafka it is important to install Kafka on our local system first. I would advice that you use docker image of Kafka, it will be a quick setup.

You can setup a basic Spring boot Project with basic spring dependencies and add the spring kafka dependecy as well i.e

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Consumer Example

@Component
public class KafkaConsumer {

    private final Logger logger = LoggerFactory.getLogger(KafkaConsumer.class);

    @KafkaListener(topics = "${spring.kafka.template.default-topic}", groupId = "group_id")
    public void consume(@Payload String protagonist) {
        logger.info("Consumed Message from show -> {}", protagonist);
        logger.info("Message received from consumer 1");
    }
}

Kafka Producer

@Component
public class KafkaProducer {

    private static final Logger logger = LoggerFactory.getLogger(KafkaProducer.class);

    private final KafkaTemplate<String,Object> kafkaTemplate;

    public KafkaProducer(KafkaTemplate<String, Object> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendMessage(String protagonist){
        try {
            logger.info("Publishing Protagonist with name: {} ", protagonist);
            kafkaTemplate.sendDefault(protagonist);
            kafkaTemplate.flush();
            logger.info("Message successfully published to kafka");
        } catch (Exception e) {
            logger.info("Failed to publish message to kafka");
            e.printStackTrace();
        }
    }

}

You can provide the Kafka configurations in the application.yaml

  kafka:
    consumer:
      bootstrap-servers: localhost:9092
      group-id: group_id
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring:
          json:
            trusted:
              packages: '*'
    producer:
      bootstrap-servers: localhost:9092
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    template:
      default-topic: hello.kafka

That is all for this blog.

Checkout our other blogs on java and java on Knoldus blogs.

Checkout our templates here.