Using eBPF probes to trace a memory leak in Rust

I was trying to find the source of a memory leak that turned out to be quite hard to pin down. My first instinct was to use heaptrack, but there was an incredible amount of memory churn and no clear "leak" in the technical sense, since all memory that was being allocated seemed to eventually get freed. Between the slow rate of the leak — a few megabytes an hour — and the magnitude of the churn, there was too much noise for me to get any useful information.

I found a few suggestions to try attaching eBPF probes to the process, using bpftrace. Most of the write-ups actually suggested hooking into malloc and mmap, but this would very quickly run into the same issues as trying to use heaptrack.

Fortunately, rather than hooking into allocations, we can instead hook into page faults. The first access of a fresh page always triggers a page fault, while most typical memory operations on pages that are already resident do not: hooking bpftrace up to trigger on page faults automatically filters out the memory churn!

1. Frame pointers

The binary needs to be compiled with frame pointers

which allows the probe we set up later to print call stacks. It's not typically necessary to include strip=none — since cargo does not strip symbols by default — but it's nice to be sure, otherwise the call stacks you get at the end won't have human-readable function names.

You'll want to make sure that glibc on your system was also built with frame pointers, otherwise the call stack trace may never reach any of the Rust function calls.

2. Monitoring

Once the program is running, set up a simple memory logger

to record the memory usage every 300 seconds. This is for your own sanity more than anything else, so that you're positive the memory usage is growing while you're monitoring it.

Use top to find the PID of the process, and record the allocated memory blocks at some point after startup

and again some time later

after the memory used by the process has grown. From these, you can identify any growing allocations

and make a note of their address ranges, in this case fa800000-ffe00000.

3. Attaching probes

Now, we'll attach a bpftrace probe to trigger on page faults in that region

and wait. Multiple probes can be attached to multiple regions at the same time, and you can extend the upper bound past the range of the region described in the smaps file if the leak is relatively fast.

The page fault will be logged as soon as one happens, but I recommend leaving the probe attached to collect a few page faults. Once you're satisfied, exit bpftrace with Ctrl-C and you should get a collection of stack traces triggering the page fault as well as their frequencies.

It's worth being precise that the call stack you get out of this isn't necessarily the problematic one — there are certainly memory access patterns where this might be a red herring — but this is a pretty quick and low-noise way to find strong candidates to investigate. And it may not be a memory leak, but you'll at least get a sense of what's allocating new pages.