Sponsor: Do you build complex software systems? See how NServiceBus makes it easier to design, build, and manage software systems that use message queues to achieve loose coupling. Get started for free.
You secure sensitive data when stored in a database, but are you securing sensitive data in an event or message driven architecture? For example, if you need to include Credit Card information in a message for a queue to be processed. But need to be compliant with PCI-DSS! You can encrypt the actual message or a portion of the message, or if the broker supports it, use server-side encryption.
YouTube
Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.
Transfering Senstive Data
When not using a message broker and doing everything in a synchronous request/response, we can simply use a secure transport such as HTTPS/TLS. As an example, if we had a checkout process where we needed to process a credit card, we would have the browser send the data over HTTPS/TLS to our HTTP API service, and then it would, in turn, make an HTTPS/TLS request to the Payment Gateway we’re using to process credit card transactions.
Our HTTP API just becomes a proxy to relay that information to our payment gateway. At no point are we storing that data on any type of durable storage where it will be left unencrypted in plain text. Everything is secured transferring it from our Client/Browser to our HTTP and then finally to the payment gateway all over HTTPS/TLS.
Securing Sensitive Data At Rest
This works fine until you actually need to persist sensitive data, or in this example, credit card info. It may not be ideal to be bound to process the credit card via the Payment Gateway when we place an order. If the payment gateway is unavailable or causes an error, we will lose the order.
Rather what we can do is place the order and then send a message to a queue to process the credit card asynchronously in a separate process.
This decouples the two actions of placing an order and processing the credit card.
So what’s the issue? It may not seem obvious, because we are well aware of securing sensitive data in a database, however, in order to do this, we must place the credit card information inside our message onto the queue, which is likely using durable storage. Now we have sensitive data that are unencrypted in plain text. Not ideal.
Encrypted Message (or Properties)
One solution to this is to encrypt the message or properties within the message before they are actually sent to the broker. When the client sends the data over HTTPS/TLS, we then create our message encrypt it (or properties within the message)
Then our consumer can pick up the encrypted message from the message broker, decrypt it locally in memory and get out the values.
This requires the producer and consumer to share a key that is used for encryption and decryption.
As an example, NServiceBus enables securing sensitive data by encrypting properties within a message. You can configure this by using the EnableMessagePropertyEncryption on the endpoint configuration.
Then it’s as seamless as using the EncryptedString type in any message. NServiceBus will handle the encryption and decryption automatically.
Here’s a screenshot of a PlaceOrder command using RabbitMQ. You can see the CreditCard Number has an EncryptedValue. Everything is stored in rabbit as an encrypted value and nothing is in plain text.
Server Side Encryption
Another option is using a message broker that supports server-side encryption (AWS SQS, Azure Service Bus). This means that all messages are encrypted at rest by the broker itself.
This means that nothing has to change in your application in terms of encrypting messages. We send the message broker our messages in plain text.
However, the broker itself is encrypting the messages while at rest.
When a consumer receives a message, the broker will unencrypt it and send the consumer the message in plain text.
All communication is done over secure channels.
Securing Sensitive Data
Often times messages are overlooked or not thought of as persistent storage or somewhere to think about having to secure sensitive data. If you have requirements to store sensitive data for various compliance reasons, make sure to secure messages either by encrypting the message at self prior to sending it to the broker or by leveraging a broker that supports encryption at rest.
Source Code
Developer-level members of my YouTube channel or Patreon get access to the full source for any working demo application that I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.