Introduction
After writing the Kubernetes (K8s) CPU Limits post, I was wondering what happens when a K8s memory limit is set for the same service. I’ve been hearing at workshops and at Ardan how people are experiencing Out Of Memory (OOM) problems with their PODs when setting K8s memory limits. I’m often told that when an OOM occurs, K8s will terminate and restart the POD.
I wanted to experience an OOM, but I was also curious about three other things.
- Is there a way to identify the minimal amount of memory a Go service needs to prevent an OOM?
- Does setting the Go runtime GOMEMLIMIT variable to match the K8s memory limit value have any significant effect on performance?
- How can I effectively use K8s requests, limits, and GOMEMLIMIT to possibly prevent an OOM?
This post is an experiment and my results may not correlate with your services. I will provide the data I’ve gathered and let you decide how relevant these findings are to your own services.
Before You Start
If you want to understand more about how the GC works in Go, read these posts.
This post is provided by Google to help with understanding requests and limits.
Creating an OOM
I need to force an OOM to occur when running the service. I don’t want to do this randomly by using some ridiculously low amount of memory. I want to find an amount that runs the service on the edge of an OOM, where it eventually happens. So I need to know how much memory the Go service is using when running a load without any knobs set. I can identify this amount by using Go memory metrics.
Luckily I have the expvar package already integrated into the service. This package provides the memory stats and I can use curl to access the expvar endpoint to read the stats once the service is running.
Listing 1
$ curl localhost:4000/debug/vars/
Listing 1 shows how I will access the Go memory values using curl. I mapped this endpoint manually in the service project inside the debug package.
There are more than a handful of memory stats that are available, but I think these three are the best ones to help understand the amount of memory the service is using when running a load.
HeapAlloc: The bytes of allocated heap objects. Allocated heap objects include all reachable objects, as well as unreachable objects that the garbage collector has not yet freed. Specifically, HeapAlloc increases as heap objects are allocated and decreases as the heap is swept and unreachable objects are freed. Sweeping occurs incrementally between GC cycles, so these two processes occur simultaneously, and as a result HeapAlloc tends to change smoothly (in contrast with the sawtooth that is typical of stop-the-world garbage collectors).
HeapSys: The bytes of heap memory obtained from the OS. HeapSys measures the amount of virtual address space reserved for the heap. This includes virtual address space that has been reserved but not yet used, which consumes no physical memory, but tends to be small, as well as virtual address space for which the physical memory has been returned to the OS after it became unused (see HeapReleased for a measure of the latter). HeapSys estimates the largest size the heap has had.
Sys: The sum of the XSys fields below. Sys measures the virtual address space reserved by the Go runtime for the heap, stacks, and other internal data structures. It’s likely that not all of the virtual address space is backed by physical memory at any given moment, though in general it all was at some point.
Start The System
I will start by running the service with no K8s memory limit. This will allow the service to run with the full 10 GiGs of memory that is available. If you didn’t read the K8s CPU limits post, I am running K8s inside of Docker using KIND.
Listing 2
$ make talk-up
$ make talk-build
$ make token
$ export TOKEN=<COPY-TOKEN>
$ make users
Output:
{"items":[{"id":"45b5fbd3-755f-4379-8f07-a58d4a30fa2f","name":"User
Gopher","email":"user@example.com","roles":["USER"],"department":"","enabled":true,"da
teCreated":"2019-03-24T00:00:00Z","dateUpdated":"2019-03-24T00:00:00Z"},{"id":"5cf3726
6-3473-4006-984f-9325122678b7","name":"Admin
Gopher","email":"admin@example.com","roles":["ADMIN"],"department":"","enabled":true,"
dateCreated":"2019-03-24T00:00:00Z","dateUpdated":"2019-03-24T00:00:00Z"}],"total":2,"
page":1,"rowsPerPage":2}
In listing 2, you can see all the commands I need to run to bring up the K8s cluster, get the sales POD running, and hit the endpoint I will use in the load test.
Now I can look at the initial memory stats by hitting the expvar endpoint.
Listing 3
$ curl localhost:4000/debug/vars/
{
"goroutines": 13,
"memstats": {"Sys":13980936,"HeapAlloc":2993840,"HeapSys":7864320}
}
Memory Amounts:
HeapAlloc: 3 MiB
HeapSys: 8 MiB
Sys: 13 MiB
Listing 3 shows the initial memory stats. The amount of live memory being used by the heap is ~3 MiB, the total amount of memory being used by the heap is ~8 MiB, and at this point the service is using a total of ~13 MiB. Remember these values represent virtual memory and I have no idea how much of that is currently backed by physical memory. I can assume a majority of the Sys memory has physical backing.
Force an OOM
Now I want to run a small load of 1000 requests through the service and look at the memory amounts again. This will show me how much memory the service is using to handle the load.
Listing 4
$ make talk-load
Output:
Total: 33.7325 secs
Slowest: 1.2045 secs
Fastest: 0.0069 secs
Average: 0.3230 secs
Requests/sec: 29.6450
Total data: 481000 bytes
Size/request: 481 bytes
In listing 4, you can see the results of running the load through the service. You can see the service is handling ~29.6 requests per second.
How much memory has been used to handle those requests?
Listing 5
$ curl localhost:4000/debug/vars/
{
"goroutines": 13,
"memstats": {"Sys":23418120,"HeapAlloc":7065200,"HeapSys":16056320}
}
Memory Amounts:
HeapAlloc: 7 MiB
HeapSys: 16 MiB
Sys: 23 MiB
In listing 5, you can see the memory amounts. The amount of live memory being used by the heap is ~7 MiB, the total amount of memory being used by the heap is ~16 MiB, and at this point the service is using a total of ~23 MiB. This increase in memory usage is expected since the service processed 1000 requests with a total data size of 481k bytes and needed to access a database. I don’t expect these memory amounts to change much as I continue to run load.
This tells me if I set a K8s memory limit that is less than 23 MiB, I should be able to force an OOM. It’s hard to tell how much physical memory is actually backing that 23 MiB of virtual memory. To not waste your time, I tried a few numbers below 23 MiB and I reached my first OOM when I used 17 MiB.
Listing 6
containers:
- name: sales-api
resources:
limits:
cpu: "250m"
memory: "17Mi"
In listing 6, I’m showing you how I set the K8s memory limit to 17 MiB in the configuration. Once I made this change, I needed to apply it to the POD and then check that the change was accepted.
Listing 7
$ make talk-apply
$ make talk-describe
Output:
Restart Count: 0
Limits:
cpu: 250m
memory: 17Mi
In listing 7, you can see the call to apply and the output of the describe command. You can see the K8s memory limit of 17 MiB has been applied. Now when I run the load, the OOM occurs.
Listing 8
$ make talk-load
$ make dev-status # Monitoring Status For Restart
$ make talk-describe
Output:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Restart Count: 1
Limits:
cpu: 250m
memory: 17Mi
Last Memory Amounts:
HeapAlloc: 7 MiB
HeapSys: 16 MiB
Sys: 23 MiB
In listing 8, you see the results of running the load with a K8s memory limit of 17 MiB. K8s is reporting the last state of the POD was Terminated with a reason of OOMKilled. Now I have an OOM, but it took 6 MiB less than the Sys number. I think looking at the Sys amount is the best amount to look at since it represents the total amount of virtual memory the Go service is using. Now I know that the service needs ~18 MiB of physical memory not to OOM.
I was curious what the performance of the service would be if I used K8s memory limits of 18 MiB (Minimal Number), 23 MiB (Go’s Number), 36 MiB (2 times minimum), and 72 MiB (4 times minimum). I think these amounts are interesting since they represent reasonable multiples of the minimum amount and by using Go’s calculated amount, I can compare the runtime’s decision against my own.
I was also curious what the performance of the service would be if I gave the K8s memory limit amount to the Go runtime. I can do this by using the GOMEMLIMIT variable.
Listing 9
env:
- name: GOGC
value: "off"
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
In listing 9, you see how to set the GOMEMLIMIT variable to match the K8s memory limit amount. K8s provides the limits.memory variable which contains the value I set in the YAML from listing 6. This is nice because I can change the amount in one place and it will apply for both K8s and Go.
Notice that I turned the GC off. When setting GOMEMLIMIT you don’t need to do this, but I think it’s a good idea. This tells the GC to use all of the memory assigned to the GOMEMLIMIT. At this point, you know what the memory constraint is, so you might as well have Go use all of it before it starts a GC.
Here are the results when using K8s memory limits of 18 MiB, 23 MiB, 36 MiB, and 72 MiB with and without the GOMEMLIMIT value set.
Listing 10
No Knobs: 23 MiB (Go’s Number)
Total: 33.7325 secs
Slowest: 1.2045 secs
Fastest: 0.0069 secs
Average: 0.3230 secs
Requests/sec: 29.6450
Limit: 18 MiB (Minimal) With GOMEMLIMIT: 18 MiB
Total: 35.1985 secs Total: 34.2020 secs
Slowest: 1.1907 secs Slowest: 1.1017 secs
Fastest: 0.0054 secs Fastest: 0.0042 secs
Average: 0.3350 secs Average: 0.3328 secs
Requests/sec: 28.4103 Requests/sec: 29.2380
Limit 23 MiB (Go’s Number) With GOMEMLIMIT: 23 MiB
Total: 33.5513 secs Total: 29.9747 secs
Slowest: 1.0979 secs Slowest: 0.9976 secs
Fastest: 0.0029 secs Fastest: 0.0047 secs
Average: 0.3285 secs Average: 0.2891 secs
Requests/sec: 29.8051 Requests/sec: 33.3615
Limit 36 MiB (2*Minimal) With GOMEMLIMIT: 36 MiB
Total: 35.3504 secs Total: 28.2876 secs
Slowest: 1.2809 secs Slowest: 0.9867 secs
Fastest: 0.0056 secs Fastest: 0.0036 secs
Average: 0.3393 secs Average: 0.2763 secs
Requests/sec: 28.2883 Requests/sec: 35.3512
Limit 72 MiB (4*Minimal) With GOMEMLIMIT 72 MiB
Total: 34.1320 secs Total: 27.8793 secs
Slowest: 1.2031 secs Slowest: 0.9876 secs
Fastest: 0.0033 secs Fastest: 0.0046 secs
Average: 0.3369 secs Average: 0.2690 secs
Requests/sec: 29.2980 Requests/sec: 35.8689
In listing 10, you can see all the results. I was happy to see better performance when I told the Go runtime how much memory was available to use. This makes sense since the Go runtime is able to use more memory than it’s using on its own.
What’s interesting is Go’s number of 23 MiB seems to be the right number when not setting GOMEMLIMIT to match. It shows how amazing the GC and the algorithms are.
When setting GOMEMLIMIT to match the K8s memory limit, using 36 MiB was a bit faster, with an extra ~2 requests per second. When I use 72 MiB, the performance increase is insignificant.
Preventing an OOM with GOMEMLIMIT
I was also curious if GOMEMLIMIT could be used to prevent an OOM. I started playing with the idea and had some success, however I quickly came to this conclusion - If the service doesn’t have enough memory, you’re not helping the service by trying to keep it alive.
I was able to keep the service running without an OOM at 13 MiB using GOMEMLIMIT.
Listing 11
Limit: 13 MiB With GOMEMLIMIT
Total: 105.8621 secs
Slowest: 3.4944 secs
Fastest: 0.0154 secs
Average: 1.0306 secs
Requests/sec: 9.4462
Memory Amounts
"memstats": {"Sys":14505224,"HeapAlloc":2756280,"HeapSys":7733248}
HeapAlloc: 3 MiB
HeapSys: 8 MiB
Sys: 15 MiB
If you look at the performance, the service is running at 9.4 requests per second. This is a major performance loss. The GC must be over-pacing and causing the service to spend a lot of time performing garbage collection instead of application work. If memory has to be kept to a minimum, I think it’s best to find the amount of memory where you live on the edge of an OOM, but never OOM.
Conclusion
Something that keeps bothering me is that I know exactly what is running on the node and I know the node has enough memory to accommodate all the services. Without that guarantee, I’m not sure anything I’m testing in this post is relevant.
If you’re experiencing an OOM, it’s possible that the node is over-saturated with services and there isn’t enough memory on the node to accommodate everything that is running. At this point, it won’t matter what the K8s memory limit settings are since there isn’t enough physical memory to meet the demand.
With this in mind and all the other things I’ve shared in the post, I have these soft recommendations.
If you don’t need to use K8s memory limits, don’t. Use CPU limits to decide what services are running on what nodes. This allows the Go services to use the amount of memory it needs. We saw in the post that Go is good at finding a low amount of memory to work with. If you’re running services written in different languages on the same node, I still feel good about not setting K8s memory limits. If you end up with an OOM, then you know there isn’t enough physical memory on the node.
If you’re not going to use K8s memory limits, then don’t do anything with GOMEMLIMIT. The Go runtime is really good at finding the sweet spot for your memory requirements.
If you’re going to use K8s memory limits, then set the request value to match the limit. However, all services on the node must do this. This will provide a guarantee there is enough physical memory to meet all the K8s memory limit requirements. If you end up with an OOM, then you know the service doesn’t have enough physical memory.
If you’re going to use K8s memory limits, then you should experiment with GOMEMLIMIT and set it to match the K8s limit amount. Obviously you need to test this for your service and load, but I think it’s worth trying. You paid for the memory and you know it’s assigned to this service, why not use all of it.
One caveat of having GOGC off and using GOMEMLIMIT. In this scenario, the GOMEMLIMIT number becomes the point when a GC starts. You might need the GOMEMLIMIT number to be some percentage smaller than the K8s memory limit so the GC starts before the K8s limit is reached. An OOM could occur if the full amount of virtual memory being used by Go is being backed by physical memory at the time of the GC. However in my experiments, I didn’t have a problem with the two settings being the same.