KSQL and Registered topics
I treat KSQL as a wrapper around Kafka Streams. If my ETL app isn’t complex and can be handled with KSQL, I prefer it that way.
In my current project a typical Kafka Streams application consists of loading the data via Kafka Connect into a topic, transforming it and putting it back into another topic. Until I discovered KSQL it was a pretty repetitive approach:
- write an Avro schema,
- generate the corresponding Java classes with the gradle-avro-plugin,
- provide Serdes and
- implement an HTTP endpoint to monitor the health of the app.
Putting together this boilerplate consisting of just a few classes, I could start implementing the main transformation logic, a.k.a. business value 💰.
As already said, in some cases, KSQL is a perfect choice to replace all of that with a few statements, like:
The first KSQL command creates a stream from input_topic. The second one creates another stream backed by output_topic.
The interesting part is revealed, when we display these topics:
All columns make sense to me besides one: Registered. It has never caught my attention, but this flag marks a topic as registered with KSQL. The topic’s information is extended by its metadata like format of the data held inside. Also, KSQL allows to drop such a topic:
They key thing to understand here is that drop topic does not delete a topic. It de-registers it in KSQL, but leaves the topic on the Kafka broker as is. You still can consume messages either in KSQL via print or with a regular Kafka consumer.
Why should you care and how did I get here anyway?
When KSQL replies with “No topic with name XYZ was registered”
I had created a stream in KSQL:
Then I wanted to delete it. First I terminated the query:
Then the stream, but accidentally instead of typingksql> drop stream xyz;
I wroteksq> drop topic xyz;
Keep calm and continue dropping, I thought.
Let’s get rid of the stream, this time properly:
Whooops… and now what? 😱
As explained above, there is still this XYZ topic on the Kafka broker, just no longer registered with KSQL. You need to register the topic again:
Now it’s finally possible to delete the stream:
I’m not sure why you have to first terminate a query until you’re allowed to drop a stream, but on the other hand you do not have to drop a stream until you can drop, or rather de-register, the backing topic. Can’t wait to be surprised by KSQL again. Like with it not being able to handle decimal types encoded as bytes. But that is another funny story very well explained by Robin.
Looking for Scala and Java Experts?
We will make technology work for your business. See the projects we have successfully delivered.
