Want to understand what to consider when choosing Event Streaming and Messaging for your Architecture?
They both are heavily used when it comes to Architecture, they are fantastic ways to manage the interactions, communication and behaviour between your services or systems and often there is not a one size fits all approach you should take.
Just because you have taken one of these approaches in a previous solution does not mean that it the right approach for your next, hopefully this post will be able to give you a high level overview on what you should consider and help you to mange your Architecture decision.
Before you start
About this post:
- 10 – 20 min average reading time
- Suitable for beginner through to advanced
What you will gain reading this post:
- An understanding of the differences and use cases for choosing the different approaches.
- Understand the cloud technologies available and which to choose.
What you can do to help support:
- Like, comment and share this article
- Follow this blog to receive notifications of new postings
Now, let’s get started.
Okay, what terminology do I need to understand before progressing?
What is an Event?
- An event encapsulates a change in state (what has happened)
- It should be lightweight and carry only enough information required about the change in state
- An event is distributed to notify any interested parties
- Events are distributed through channels such as streaming and messaging
An example, when a consumer borrows a DVD, the DVD state changes from “for rent” to “rented”.
What is a Message?
- A message encapsulates the intention / action (what has to happen)
- It is not lightweight containing a payload of all data required for processing
- A message can be formatted and adhere to a contract to be suitable for the interested parties
- A message is distributed through channels such as messaging
An example, when a customer places a reservation on a DVD but requests it be collected from another location, the booking system receives a message to reserve the stock and trigger a transfer for the DVD to another location.
Isn’t Events and Messages interchangeable?
A message is often interchanged with the term Event, however it is intended for different purposes even though there are many commonalities between them.
Referring to the CQRS pattern, checkout the Command message (message) and Event message (event) which demonstrates the differences between them.
What is Event Sourcing?
- Event sourcing is a way to atomically update state and publish events.
- Event sourcing is about a business object being persisted by storing a sequence of change in state events, that when replayed will show the entire state of the business object, rather than saving only the current state.
An example is a Payment Service.
- It stores a Payment by persisting its change in state events – Created, Paid, Cancelled.
- Each event would contain sufficient data to reconstruct the Payment’s state.
What is a Stream?
- A stream consists of immutable data, only inserting new events, whereas existing events cannot be changed.
- Streams are persistent, durable and fault tolerant.
What is Streaming?
- Streaming of data is the constant flow of events where each event should contain enough information to reflect the change in state.
- It allows for the processing of data to occur in real-time (data in motion) and is different from the traditional approach for the processing of static data to occur (data at rest) at a later point in time, known as batch processing
- Streaming data is unbounded, meaning it has no real beginning and no real end
- Each event is processed as it occurs and is managed accordingly
An example of this can be found in the stock market.
- When a stock price change, a new event is created containing the time and day, the stock identifier, and its new trade price which in this example is the change in state
- Given there are thousands of stocks, and thousands of trades happening every second, this results in a constant stream of data
What is Publisher (pub)?
- An application that publishes a message to a topic
- It does not know the destination of the message
What is Subscriber (sub)?
- An application that registers itself to a topic to receive the messages
- It does not know the source of the message
What is a Queue?
- A Queue is a mechanism for system to system communication that is asynchronous in behaviour
- Queues store messages until they are processed and deleted.
- Each message is processed only once and for a single consumer.
- Queues can be used to decouple heavyweight processing, to control the flow of an influx or batch data, and to support erratic workloads.
- This is typically used in Serverless and Microservices architectures.
What is Topic?
- Is a channel that maintains a list of subscribers to relay messages to that are received from publishers
Great, now the terminology is clear, let’s explore these in greater detail
Why use Event Streaming?
Event streaming is really powerful when you have events that you want to be able to process and perform analysis in real-time allowing your systems to immediately take action.
| Use Case |
|---|
| Processing – Payments – Stocks |
| Detection – Fraud – Anomaly |
| Maintenance – Predictive |
| Analytics – IoT |
- Event streaming is used for real-time processing and analysis of changes in state in the data through events
- It can persist events supporting the rebuilding of the state through replaying events in the order they were received
- It allows multiple consumers to receive each event
What Cloud services are available?
Depending on the cloud provider you are using the options are quite extensible and depending on your use case the suitable technology choice may differ.
Below is a simplistic decision flow to help you determine which technology may be more suitable depending on your use case, for more complex scenarios it would be worth asking yourself more questions to determine if the particular technology choice will be suitable.
Why use Messaging?
Messaging is powerful when it comes to decoupling your systems and providing a highly available and durable solutions, apart from a few use cases, messaging is more of a way to manage your system interactions and control the ingestion of your data.
Pub / Sub
| Use Case |
|---|
| Applications – Stateful |
| Workflows – Asynchronous |
| Workloads – Balancing |
| Systems – Stocks |
| Notifications – SMS |
| Delivery – Multiple consumers – Ordered |
- Publish-subscribe is commonly referred to as pub-sub
- Pub-sub moves data from producers to consumers
- It allows multiple consumers to receive each message in a topic
- Publish-subscribe ensures that each consumer receives messages in a topic in the exact order in which they were received by the messaging system
Queueing
| Use Case |
|---|
| Workflows – Asynchronous |
| Workloads – Balancing |
| Systems – Pay Check – Task Lists – Work Queues |
| Delivery – One consumer – Ordering not important |
- Message queueing ensures that for exactly one consumer each message is delivered and processed
- It does not ensure that messages are delivered or processed in order
- However, each message is removed from the queue once it has been delivered but it does requires consumer acknowledgement
What Cloud services are available?
Once again, depending on the cloud provider you are using the options are quite extensible and depending on your use case the suitable technology choice may differ.
Below is a simplistic decision flow to help you determine which technology may be more suitable depending on your use case, for more complex scenarios it would be worth asking yourself more questions to determine if the particular technology choice will be suitable.
Conclusion
- Do you understand your use case?
- Do you understand how this may evolve?
- What does the roadmap for your product look like?
This will be the driver for the technology choice that you should use.
If you require…
- Processing and analysis of your data in real-time
- If you have a constant flow of data changing in state that you may want to persist, replay or use to recreate your state
Then event streaming is the path to explore.
However, if you require…
- To have decoupled, highly durable and highly available Architecture
- The execution to be asynchronous in behaviour
- If you don’t require real-time processing of your data, or if you are issuing commands
Then messaging is the path to explore.
Does that mean that you can’t combine these approaches?
Not at all, however event streaming is a much more complex setup and therefore my recommendation is if your use case won’t evolve and require real-time processing and analysis of your data, then the complexity that event streaming adds to your solution most likely is not worth it.
Did this help?
- Like, comment and share this article
- Follow this blog to receive notifications of new postings
- View previous postings
Great Article
Thank you