Profiles

APM → Profiles shows where each service spends its CPU time. Every CPU on every monitored node is sampled continuously — about 19 times a second — so there’s always a profile to look at when something gets slower, with no code changes, no agent in your images and no restart.

Reading a flame graph

Each bar is a function; the bars below it are the functions it called. The wider a bar, the more CPU time was spent in that function and what it called. The top bar under the service is the process.

• Hover a bar to see its share of the samples, and how much was spent in the function’s own code rather than in what it called.
• Click a bar to zoom in to it; click it again to zoom back out.
• Find a function highlights every bar whose name matches.
• Busiest functions below the graph lists the functions with the most CPU time of their own — usually the first place to look.

Choose a pod to see one replica on its own.

Comparing two windows

Compare with the period before sets the same length of time just before the window you’re looking at; Compare with… takes any window. Bars are then coloured by how their share of CPU time changed: red grew, blue shrank. The comparison is by share, so a busier or longer baseline doesn’t make everything look like it shrank.

On a service’s page, Compare CPU before/after next to the last deploy opens the 30 minutes after it against the 30 minutes before — the quickest way to see what a release changed.

What’s named

• Go programs are named down to the function, with no setup — Go programs carry the information needed.
• The Linux kernel — system calls, networking, scheduling — is named too, marked [k].
• Other languages (C, C++, Rust, Python, Java, Node.js) show as the library or program the code is in, with an offset, for now.

On Arm nodes, a few frames next to the innermost function can be attributed to a neighbouring function; the overall picture is unaffected.

Cost and retention

Sampling uses well under 1% of a CPU on each node and about 30 MiB of the sensor’s memory. Profiles are kept for 7 days. To turn profiling off, or change the rate, see sensor.profiling in Configuration.