Asynchronous Image Processing in Ruby on Rails with Shrine

As Ruby on Rails developers, we often encounter the need to upload and store images provided by our users. 

Often these images need to be resized and saved as multiple versions: for mobile, web, thumbnails, OG images, and other custom and non-standard formats needed by the Client.

What problem does it lead to? Image processing is a computationally intensive operation and the latency will inevitably drop.

Solution? Sidekiq + Shrine!

So… What can be done about it? If you are like me, you want both to have a cake and eat it. 

In this blog post, we will go through how to achieve fast response times while generating multiple versions of an image. 

The solution is simple. We will use Sidekiq to schedule a job for image processing and immediately return a 200 status response.

For image upload, we will use Shrine. It’s a modern library with plugin design and is an alternative to gems like Carrierwave or Paperclip. I like it very much and recommend using it in your projects - it has truly great documentation that cannot be overstated.

How does it work in practice?

Step-by-step Guide

You can see the full application on my Github.

rails new shrine-uploader-api --api --database=postgresql && cd shrine-uploader-api && rails db:create

Add the following to your Gemfile:

And run:

bundle

Let’s use a resource generator from Rails CLI. We will create an uploads table with an image_data column which is essential for Shrine to work:

rails g resource Upload image_data:jsonb && rails db:migrate

We need to configure the Shrine first. 

We will declare 2 types of storage: 

• “cache” for raw images 
• “store” for processed images. 

In this section, we will also declare plugins for integration with ActiveRecord, derivatives generation, and background processing. The last part registers a callback fired on the “promote event”. Promote event is run after our record “Upload” is persisted to the database.

Next, let’s add a Shrine uploader with specifications on derivatives we want to generate. Image processing is handled underneath by imagemagick:

Let’s plug the uploader into our model. Images now can be accessed through the image virtual field:

We need to implement a PromoteJob that we used in our initializer. This job will be put into Redis on the promotion event and later executed in Sidekiq. Here is the code for it:

The last thing is to implement a “create action” in the UploadsController.

Boom! We finished with the code!

Now let’s run the Redis server, which Sidekiq uses to store and retrieve jobs. I will use an official docker image, but you can run a Redis from your system if you prefer :)

docker run --name my-redis -d --publish 6379:6379 redis

And run sidekiq:

bundle exec sidekiq

Run Rails server on the other terminal tab:

rails s

And there you go! Our application is functional and running :) 

We have an endpoint http://localhost:3000/uploads where we can send images and they will be processed according to our specifications. 

Now let’s test it!

First, we need an image to upload. It can be any image. In my case, it’s a file I called example.png

curl -X POST -i -F image=@example.png localhost:3000/uploads

The goal here is to get a 200 response and afterward end up with four images in the public/uploads folder: the original image and 3 copies resized according to the rules from ImageUplaoder.

Also, you will see that PromoteJob has been scheduled and successfully run in Sidekiq logs.

Now, let’s see how we are doing on the performance side of things.

First, a little trick to get the response time from curl. I will create a file curl-format.txt with the following content and use it later as an argument.

time_total:  %{time_total}sWe will send our image in the binary form using a -F image=@your-file-name.png flag.

15ms, not bad ;) 

I ran some tests without using Sidekiq to asynchronously process images requests and took around 120ms. That’s 8 times slower than with background processing enabled!

Conclusion

Let’s be honest. Not every Rails application needs background image processing.

But there are situations when it is a really good idea. Imagine building an API where multiple images are uploaded every second. Or let’s assume you do very complex image transformations and it takes a lot longer than in our example.

In these cases, the delays may become noticeable and this simple trick I described above will surely help you keep up the great performance of your app.

Happy coding!