Copy

Reading New Relic's Ruby VM Tab

One of my favorite features of New Relic is the Ruby VM tab. You can access it by click "Ruby VM" in the left hand menu.

The Ruby VM tab can tell you a lot about the memory behavior of your Rails application, which can, in turn, point to different performance issues and opportunities for improvement.

However, I find many of my clients never check this tab, because they're unsure of what it means or how to turn it into action. Let's talk about each metric in turn. For each metric, we'll talk about what it means, what an acceptable range of values are, and then what you can do to fix problems with that metric.
 

A Quick Aside: transactions and web vs. nonweb


When you initially open up the Ruby VM tab, you are looking at the average Ruby VM metrics across all servers. This means you're looking at servers running Puma/Unicorn/Passenger/whatever (your web servers) and probably separate servers running background jobs (Sidekiq/etc).

These two types of Ruby processes have extremely different memory behavior. You should analyze them separately. To do this, click the "servers" dropdown in the top bar and select a single web server or a single background job server.

New Relic uses the term "transaction" to mean "one web response or background job". What exactly that term means will depend on the context - whether or not you're looking at background job servers or web servers.

Time Spent in GC

This one is pretty self explanatory - it's the total percentage of time that transactions spent waiting on GC. To get this metric, you must explictly enable it by calling `GC::Profiler.enable` anywhere in your application (probably in an initializer).

Usually, this number is less than 1%. That tends to shock people: if you've never deeply studied the performance of Ruby before, you tend to assume there's lots of time spent in GC. There actually isn't! Most applications spend time doing many other things, like running Ruby and waiting on the database.

Memory Usage

The resident set size of the Ruby processes running on this server - also known as their RAM usage.

Typical Rails applications use between 350 and 700 MB per process, but generally anything up to 1GB is not a major cause for concern.

If you're using more than 800MB per process, you can't safely use the c5 or c4 series instance types on EC2, which are my preferred high-performance instances. If you're using more than 2MB per process, you can't use the m5 series, which means you must use the inefficent and expensive r4/r5 series. Lower memory usage means higher-perf instances are available.

Ruby Heap Size

This metric tells us a bit about the state of the Ruby object space. The number of live objects is simply the number of objects currently live, and free heap slots is the number of slots available for new objects. When the number of free slots available reaches zero, garbage collection is triggered.

The heap is typically between 1-2 million slots. An extremely high number of free slots (1 million or more) is indication that you have a single transaction (or a handful of transactions) that are allocating a large number of objects at once.

Larger heap sizes than 2million objects tend to slow down garbage collection and object allocation.

GC Frequency

This metric tells us how often GCs are occurring on the server. Minor GCs tends to be fast - they check only a small subset of all the objects (just the "new" ones) and they are "interleaved" (they are a series of short pauses rather than one big long pause). Major GCs are more serious, as they take longer (up to 100ms or more) and must check all objects.

On average, I see about 10-20 GC runs per 100 transactions, with about 1-2 major GC runs per 100 transactions.

High GC frequency (>30 runs per 100 transactions) is associated with excessive GC time and excessive time spent allocating objects.

Object Allocations

Pretty simple: how many objects get created during every transaction. Remember that it's very easy in Ruby to create objects: Every string literal ("this" or "that") creates a new string object, and operations on collections (like "map") often have to duplicate the array and all of it's contents first.

On average, I see about 50,000 object allocations per transaction.

Excessively high allocations per transaction (100k or more) is associated with high latency (because you're spending so much time creating and allocating objects). Basically, it helps Ruby quickly figure out where methods are defined.

Method cache and constant cache invalidations

Here's a really in-depth definition of Ruby's method cache by Charlie Somerville.  Basically, it helps Ruby quickly figure out where methods are defined.

On average, I see about 0.1-0.2 invalidations per transaction. Numbers higher than that indicate that you or your dependencies are probably doing a lot of metaprogramming.

I don't have a clear sense of the impact this has on performance, however. I have no clear guidance at this time for apps for which this number seems unusually high.
 

Next Week: What To Do About It, and How To Build Your Own


As you've noticed, almost all of these metrics highlight the impact of excessive object allocations in a Ruby application. Next week's email will talk about figuring out what transactions are causing this issue, and then discuss how to build similar functionality to New Relic's Ruby VM tab into your own production perf monitoring setup.

Thanks for reading - talk to you next time, Rubyists!

-Nate
 
You can share this email with this permalink: https://mailchi.mp/railsspeed/whats-new-relics-ruby-vm-tab-for?e=[UNIQID]

Copyright © 2019 Nate Berkopec, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list.