Speeding up the HTTP service with Redis caching
How responses to recurring, idempotent requests could be reused to speed up your service by a great factor.
I’ve been recently working on migrating maven-badges service from Ruby to node.js + TypeScript. The decision to move was mostly based on these few facts:
- the original project was rather small and easy to rewrite;
- it’s much easier to maintain a node.js project for me and my colleagues at SoftwareMill since we don’t work with Ruby code on a daily basis;
- the application itself doesn’t need to perform any heavy computations — it mostly composes results of few asynchronous, network IO actions — and this is where node.js’ non-blocking model really shines.
Maven-badges is a relatively simple, yet quite important project. Its main role is to provide these tiny “badges” — a picture with some label and version number, as seen on GitHub repositories:
While the main “producer” of these pictures is a 3rd party service called shields.io, maven-badges is a one-stop shop for:
- fetching pieces of information about the latest (or verifying provided) version of given library from maven-central (note that maven-badges is popular among JVM-related projects);
- requesting a “badge” from shields.io based on that data and forwarding it to the user.
Clearly, the service acts as a proxy, integrating functionalities of two distinct providers.
Shields.io can be slow
The migration went smoothly. Unfortunately, during the development, I discovered that the sole act of image creation (requested with shields.io) can be very slow. In worst cases, it took up to 10 seconds (!) to get a response from that 3rd party, which — from the outside — could look like it’s our service that was laggy. Combined with an extra time spent on fetching data from maven central service, plus the geographical distance between the user and our service, I observed rather worrisome metrics on Heroku:
The median (50th percentile) was at 831ms — it means that only half of all responses were served faster than that. At the same time, it took more than 5 seconds to handle 5% of all requests. That was alarming!
Using Postman, I sent a couple of requests to both maven-central, as well as shields.io only to confirm, that while the former does not cause a problem, the latter tends to be very slow. It’s still not clear why, though — changing request parameters (like badge’s subject, version to display, color or style) did not show a significant effect on the response times. My best guess, for now, is that it’s a quite popular service under heavy load, and it seems like it doesn’t use any caching features.
Reusing cached responses
Every parameter to shields.io (eg. subject, color, style) is contained within the URL of the simple GET request, eg:
http://img.shields.io/badge/maven_central-2.1.0-brightgreen.png?style=default
The URL above describes a request for the badge shown earlier in this post — a PNG image with “maven central” label and “2.1.0” on the bright green background.
If for given URL, containing a fixed set of parameters, the response is always expected to look the same, it could be cached and later reused for subsequent queries with exactly same arguments! This way, only the first request for a previously unseen combination of params would need to make a full roundtrip to the laggy 3rd party service.
I decided to use Redis, which is an efficient, in-memory key-value store with a simple API and the ability to define an expiry time for every key-value pair. Every URL (as described above) would then become a key, while a corresponding image (binary data) — a value. In order to avoid quickly cluttering the Redis instance, every key is assigned a 12-hour time-to-live, although every time it is asked for, that time is reset. That way, I made sure that “popular” values persist in the store, while those rarely asked for — expire and free up some space. This kind of caching strategy is called “Least recently used”, or LRU for short.
The results
Adding Redis to the project was just a matter of spinning up a docker image on my local machine for development. In production environment (Heroku), it was as simple as choosing a Redis add-on for the project — it provisioned in less than a minute. I was only afraid about the storage limit in the free tier (25MB) — but it quickly turned out that the good strategy around expiry times — as described above — was really helpful. On average, ~1000 key-value pairs are stored in the cache, which occupy less than 3MB of space.
Now, for the most important metric — a drastic drop in response times:
The median dropped from over 800 down to 17ms. This is fantastic. Also, “last“ 5% of all response times went down over four times.
Summary
Even in the case of such a simple service, there is a space for improvement. This is especially true when it depends on slow, 3rd party providers, which cannot be directly controlled. Requests that are expected to always resolve with same output (additionally being idempotent — meaning that they don’t modify any resources or application’s state) could indeed be sped up with cached results.
