Investigations

When an alert fires, an SLO burns or a monitor fails, the first question is why. An investigation answers it from your own telemetry: what degraded, what changed just before, and what that points to, with every finding linked to the view that shows it.

Click Investigate on a firing alert (on the Alerts page), a burning SLO, a synthetic monitor or a service page. Or open Monitoring → Investigations and pick a service. Anyone who can sign in can investigate; it only reads what they can already see.

What it looks at

An investigation compares the time since the problem started (from a few minutes before) with the hour before it. It looks at the service in question, or, for an alert that covers every service, the ones that degraded most.

What happened:

• how its errors, latency and traffic shifted;
• the routes that degraded most;
• callers that were affected;
• other alerts and SLOs that started around the same time.

Likely causes, most likely first:

• Deploys of the service or of anything it calls, especially just before the problem started;
• Kubernetes changes: replicas missing or scaled to zero, containers restarting;
• Dependencies that degraded: the services it calls, from your traces;
• New error logs: error messages that weren’t there before, or are surging, with repeats grouped together;
• Failing synthetic monitors.

The hour before is split in two, and the healthier half is used for each measure. That way a short incident earlier in the hour doesn’t make this one look normal.

If nothing degraded, the investigation says so rather than guessing, and mentions anything notable elsewhere as worth knowing.

Past investigations

Every investigation is kept for 180 days, with who ran it and when. Reopening one shows what was found at the time; Run again checks afresh.

AI summaries

Investigations need no AI model. If your administrator connects one, each investigation also gets a short written summary: what is wrong, the most likely cause and how confident it is, next steps, and what the evidence can’t tell you. It is written within seconds of the investigation and stored with it.

• Your choice of model: one you host yourself (for example with vLLM or Ollama), or a hosted one: any OpenAI-compatible service, Anthropic or Gemini. See AI summaries to set it up.
• Only the evidence is sent: the findings and numbers on the investigation page, never your raw telemetry. Administrators can also keep log lines out of it.
• Every summary is in the audit log (category Data sent out (AI)), with who asked, the model, where it was sent and how many tokens it used.
• The evidence comes first. The summary is a reading of the evidence and can be wrong; the findings below it are the source of truth.

Older investigations get a summary only when you click Write a summary, so browsing history doesn’t use your model.

Follow-up questions

Under the summary you can ask about the investigation in your own words — “did redis-cart restart?”, “which endpoints got slower, and by how much?”, “what do the payment service’s error logs say?”. The model answers from the evidence and, when that isn’t enough, looks things up for itself: a service’s health, its endpoints, what it calls, its error logs, its deploys, Kubernetes workloads, and metrics.

• It only reads. The lookups can’t change anything, and they’re limited to this investigation’s site and its two windows: the problem and the hour before it.
• Each answer shows what was looked up, so you can check it.
• Each question has a limit of lookups, so one question can’t run up a long bill or load the system.
• Every question is in the audit log, like summaries, with what was looked up and how many tokens it used. When your administrator has chosen to keep log lines from the model, the log lookup returns how often each kind of error happened, never the lines themselves.

Earlier questions and answers on the same investigation are kept, and the model sees them when you ask the next one.