consultancy speaking content contact
LED panels showing logs

100x'ing bulk import speed in Rails with gnarly SQL

Processing masses of data is always tricky, and it's tempting to reach for specialised tools the moment we hit problems like this, but it's my firm belief that pushing forward with the tools you're already using has compounding benefits.

In the above presentation, we deep-dive on some Rails and PostgreSQL APIs and features which allowed me to optimise bulk imports in a large-scale Rails application down to less than 1% of it's original processing time.

If you'd like to follow along in code, the exercise is available on Github – you can fork it to your own account and push changes to see the performance impact in Github Actions. It could make a good training exercise for your team, or even to pair on in an interview.

The final optimised version is also available on Github. You'll see this takes import time from 25 seconds down to a few hundred milliseconds, memory down from 10 megabytes to less than half a megabyte and allocations from over 140,000 to less than 7,000 – this gives enormous scope for scalability.

Hero Image Credit: Leif Christoph Gottwald

Video Content @LessonsofaCTO

The Cookie Store API: a new JavaScript cookie API... in 2024?! Invoker Commands: an exciting upcoming Web Platform API you’ve probably not heard of yet The Most Important Part of Scrum Isn’t The Responsibility of Developers Interaction to Next Paint is NOT an SEO Metric Practice Precise Communication Proxy Your Images For a Better Largest Contentful Paint YAGRI: You Ain’t Gonna Regret It SPA vs MPA Showdown: Which Makes Sense? Stop Littering the Web with Social Media Icons React & Next.js Are Not the Answer to Life, the Universe & Everything Scoring A+ on SecurityHeaders.com Improving Your Typing Speed Is a Micro-optimisation Passing Core Web Vitals on 3G Your Customers Don’t All Own the Latest iPhone Stop Using Your Registrar for DNS Make More of Mundane Tasks You Probably Don’t Need Native Mobile Apps You Might Do Agile, But Do You Have Agility? 3 Ways to Keep Your CDN Fast, Even When Content Changes Offload over 95% of Your Origin Traffic to Your CDN You Can’t Outsource Responsibility for Performance, Security and Accessibility Sweat Your Tech Stack 'Years of Experience' Are More Guidelines Than Actual Rules Pair Programming Doesn't Half The Output Your Infrastructure Isn’t Your Differentiator Optimise Your Tests For Readability Over Conciseness Don't Be a Hero Keep a 'Brag Book' of Your Achievements

Get in touch

Sending message…

Error sending message

Message could not be delivered — try the full contact page for alternatives

Message received

Thank you for your message — I will get back to you within 24 hours.