If you'd like to destroy your apps performance silently, without warning, and with just one setting - by all means, keep tuning!
|
|
I just finished up with a client engagement. I was able to reduce their backend response times by almost 50% with just a single change. What did I do?
I deleted the RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR environment variable.
This one environment variable, which had been added in an attempt to reduce memory usage, had cost this company a near-doubling of their Rails app's response times. Could any possible memory savings be worth that kind of price? And this isn't even the first time I've had to fix performance by deleting old GC tuning that someone had added and promptly forgotten about - this happens all the time!
In case you weren't aware, several parameters that Ruby's GC uses to make decisions about "when to GC" are available for tuning via a series of "RUBY_GC_*" environment variables. They're pretty easy to find if you look through gc.c. Back in the Ruby 2.0 -> 2.3 days, there was something of a fad of tuning these variables for various ends. Someone launched a Heroku addon that promised to tune your GC via some spooky algorithm, and various blog posts and tweets extolled the virtues of changing this or that. Well, I'm here to tell you 99% of that advice is out of date, and 90+% of apps should not use any of these environment variables.
Why tune GC?
The first question we should be asking ourselves is why are we messing with this stuff at all? Is this a good idea?
There are only two reasons to ever consider tuning these GC variables:
- Reduce maximum RSS/memory usage (usually involves changing variables that end in GROWTH_FACTOR)
- Speed up application boot (usually involves changing initial values, like GC_INIT_SLOTS)
The first reason is bad and outdated advice which can torch your app performance, like my client's, if implemented incorrectly. The second reason is fine, and we'll touch on it again at the end, but the impact is so limited that I'm hesitant to recommend everyone try it out.
When you start a Ruby process, it starts with a very small area of memory which then grows with each GC. (Almost) every time you GC, the heap gets a little bit bigger in various ways, which delays future GCs and keeps the heap only "as big as it has to be", keeping memory usage down at the cost of some performance.
Reducing memory usage through tuning GC variables was based around the idea that GCing more frequently as you grew the heap would mean that the heap was less fragmented. Fragmentation causes increased memory usage, a little bit like how Windows 95's disk defragger could give you some free space on your disk. Fragmentation increases total usage by making the heap look like swiss cheese rather than mozzarella: lots of little pockets of empty space, causing it to take up a greater volume.
Unfortunately, this promise of more-GCs-equals-less-fragmentation did not actually materialize. It never worked very well, if it ever worked at all, and mostly only served to make people's apps boot slower. This is similar to Out-of-Band GC in Unicorn, which Aaron Patterson removed at Github after he discovered it accounted for 10% of all CPU usage. GCing more frequently may reduce memory usage (sometimes!), but it can come at an unacceptable cost.
These days, reducing memory usage in Ruby can be done to much better effect (and even sometimes speed up the app) by using either jemalloc as your memory allocator or using Hongli Lai's malloc_trim patch. Using either of these approaches gives better results than GC tuning ever did, and there's no way it can tank your app's performance. It's simply better.
How do we tune GC incorrectly?
GC in Ruby involves several limits - when those limits are crossed, GC occurs, and if those limits are still exceeded after all the dead objects are freed, then we must grow the heap to accomodate. When we do that, we multiply the current limits by their corresponding GROWTH_FACTOR and grow the heap by that amount.
For example, when you start a Ruby process, it starts with 10,000 slots for objects (RUBY_GC_HEAP_INIT_SLOTS). Eventually, you will create enough objects that we can't mark any as free, and we must grow the heap to give you more slots for new objects. The heap grows by a factor of 1.8 (set by RUBY_GC_HEAP_GROWTH_FACTOR), so we'll create 8,000 new slots for objects. Now our heap has 18,000 slots (10,000 used, 8,000 free).
In the specific case of my client, they had changed RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR from its default of 2 to 0.9. This setting is helpfully documented:
"Do full GC when the number of old objects is more than R * N, where R is this factor and N is the number of old objects just after last full GC."
So, when this factor is set to 0.9, that means that old object limit was constantly decreasing (all the way to zero, eventually). This caused full GCs to happen in about every 1 in 2 requests. Ouch.
Unfortunately, this exact advice and this exact factor are currently still advised to users of fluentd. I even found an old Gitlab thread where they almost set this variable to 0.9... bet they're glad they didn't.
What GC tuning still makes sense?
I mentioned earlier that GC tuning can still be useful in one circumstance: reducing app boot time. This is true.
In local or dev environments, messing with the initial values of Ruby's GC can provide a small reduction (1-10%) in boot time. If set to the correct value, this comes at zero memory cost and does not affect the process after boot.
Try booting an irb console with $ irb, then check GC.stat. Note the number of GC cycles (count) and the number of heap_available_slots. Now, restart your irb console with the environment variable for the initial number of slots to be equal to the number you observed.
For example, when I start irb, I see there's 36000 available slots. I see 19 GCs have occurred (4 major and 15 minor). If I start an IRB session with $ RUBY_GC_HEAP_INIT_SLOTS=36000 irb, GC.stat shows just 5 GCs (4 minor 1 major). Neat!
There are several limits in Ruby's GC which can trigger major or minor GCs. These days, we've documented all of them, so you can read about them here. Simply changing RUBY_GC_HEAP_INIT_SLOTS to its value after your app boots though usually eliminates 80% of the GC cycles.
However, even this variable is not without dangers - set it too high (higher than your app will ever need), and you'll see increased memory usage.
Delete your GC tuning
To be honest, I'm expecting to be tweeted and emailed a bunch of screenshots of downward-sloping Datadog and NewRelic graphs after this newsletter - do share yours if this helped you!
Until next time,
-Nate
|
|
|
|