Years ago, a colleague of mine read an article on Google's bitpacking algorithm for storing indexes and an idea was born to create a standalone text index and search solution (similar to Apache Lucene). This idea was brought to life in a C++ application called Zsearch. In hindsight it was a little bit ahead of its time as a few years later we saw the emergence of a more powerful Elasticsearch that did all that Zsearch could do + a lot more. Anyways, this is a story of how we hunted down a memory leak in our application.
The idea behind Zsearch was to see how big of an index we could fit into ram with the data being stored in a key/value database, which in our implementation was leveldb. Since RAM was a precious commodity back in the day, we really wanted to find the right balance between keeping all of our data structures in RAM vs. leveldb. During our initial load testing on a linux box, we found a linear memory growth and therefore suspected a memory leak. We then tried to find a memory leak using the following:
- running a load test in a function and calling the function multiple times to test memory usage
- logging in destructors to check whether objects were being deallocated
- debugging the application using valgrind
Unfortunately for us, there were no memory leaks detected and eventually when testing on a mac, we found that memory stabilized at the expected peak. We therefore concluded that it was just the linux memory allocator that was acting weird. In reality, the memory allocator was doing what it was supposed to do but it did give us a good lead regarding how some of the C++ standard library was implemented. It turns out that one of our threads was making copies of an stl Map, which is ridiculously inefficient and something that we ended up reworking to keep our memory consumption on point :).
Further reading: