Runbooks

An investigation finds what changed just before a problem: a deploy, a workload scaled down. A runbook says what fixes it. When a runbook matches an investigation, the investigation shows a suggested fix, with a button that runs it as you (see Actions).

Every alert and SLO that starts firing is investigated automatically, at most once an hour each, so the evidence and any suggested fix are waiting before anyone looks. You can still investigate anything by hand, and runbooks are matched against those investigations too.

Open Monitoring → Runbooks to see them, turn them on or off, and see what each has matched recently.

Built-in runbooks

Four ship with the product, all set to suggest their fix:

Runbook	When	Suggests
Roll back a deploy that broke its service	The service with the problem, or one it calls, deployed a new version up to 30 minutes before the problem started, nothing has deployed it since, and its own errors or p95 latency are at least 3× its baseline (errors at least 1%, or p95 up at least 50 ms, on at least 100 requests).	Roll that workload back to its previous version.
Restore a workload that was scaled down	A workload has fewer replicas than it had before the problem started.	Scale it back to the replicas it had.
Delete the one bad pod	One pod of a service has errors or p95 latency at least 3× its sibling pods’ (errors at least 5%, or p95 up at least 50 ms, on at least 50 requests), and it is still running.	Delete that pod; its controller starts a fresh one.
Drain a node under pressure	A node is short of memory, disk or process IDs and still ready, and a workload in the problem runs on it.	Drain the node: cordon it and move its pods to other nodes, respecting disruption budgets.

A node that isn’t ready gets no suggestion: Kubernetes already stops scheduling there and moves its pods off, and a drain can’t finish while the node is unreachable. The investigation still names the node as the likely cause.

Thresholds compare a service with itself, not with fixed numbers: its baseline is the healthier half of the hour before the problem. A service that is always slow doesn’t trip a latency threshold; one that suddenly gets slower does.

For a built-in you can choose where it applies (namespaces, services), set its thresholds for particular services, or turn it off. Its conditions and action come with the product, so upgrades improve them. Reset to defaults undoes your changes.

How far a runbook goes

Setting	What happens when it matches
Suggest the fix	The investigation shows the fix and a button to run it.
Propose for approval	The fix is proposed: it waits on the Fixes page, and on the investigation, for a person to approve or reject it. The alert’s channels get a notice with a link to it. See Approving fixes.
Propose, and shadow	Proposed as above, and it also records what it would have done on its own: act at once, or hold off and why. Nothing runs by itself. See Shadow mode.
Act on its own	When an alert’s or SLO’s investigation matches and no guardrail holds it, the fix runs by itself, at sites that allow it, and is checked. Otherwise it’s proposed as above. See Automatic fixes.
Record only	Nothing is shown; the match is recorded on the runbook. Use it to see how often a new runbook would fire before anyone sees it.
Disabled	Nothing.

Only a runbook set to Act on its own runs a fix by itself, and only at a site whose owner allows it. Any other suggested or proposed fix runs only when someone confirms it and the cluster allows them.

Checking a fix worked

Kubernetes saying a rollout finished isn’t the same as the problem being fixed. So after a fix runs from an investigation, approved or suggested, the service the investigation was about is watched until it’s back to normal. So is the service the fix changed, when that’s a different one.

• Back to normal means what the investigation would call no significant change from the service’s baseline: errors no more than a point above it, p95 latency no more than 20% (or 20 ms) above it, and traffic not down by half or more. The baseline is the one the investigation measured, from before the problem.
• Checking starts a minute after the change finishes, when the old pods’ traffic has gone, and reads the last three minutes each time. Back twice in a row is Fixed.
• Still not back at the runbook’s verify.within (10 minutes unless it says otherwise) is Didn’t fix it. If the change itself failed, that counts too. The alert’s channels get a fix didn’t work notice, and the runbook doesn’t suggest or propose that fix for that target again for an hour, so someone looks rather than retrying the same thing.
• Too few requests to judge (under 20 in a check) at the deadline is Couldn’t tell.

The result shows on the fix, on the investigation and on its root-cause write-up, and goes in the audit trail under Actions. A runbook with no verify is checked by Kubernetes only, and so is an action run from a workload’s Actions menu, which has no investigation to compare with.

Approving fixes

A runbook set to Propose for approval turns each match into a proposed fix. Open Monitoring → Fixes, or follow the link in the notification, to see what it would do and why, who it came from, and how long it waits.

• Approve shows what the fix will change, asked of the cluster as you, and then runs it as you: the cluster’s permissions apply and its audit log names you, as with any action. Approving needs the editor role and a recent sign-in.
• Reject says no, with an optional reason for whoever looks next. Nothing changes on the cluster.
• A proposal nobody decides on expires, 30 minutes after it was made unless the runbook says otherwise (expires, 5m to 24h): by then the evidence it rests on is stale. Investigate again if the problem is still there.
• Before a fix runs, its target is checked against what the investigation saw. If someone has already rolled the workload back, deployed it again or scaled it by hand, or the pod is gone, the proposal is superseded and nothing runs.
• With approvals: 2, two people must approve, and the person who asked for the investigation can’t be either of them. The first approval only checks that the cluster would let them; the fix runs as the second.

A later investigation of the same problem joins the open proposal instead of adding another. Approvals, rejections and the action they run are in the audit trail under Actions.

The notice goes to the channels of the alert or SLO behind the automatic investigation, with state proposed and a link to the fix, except PagerDuty and incident.io, where it would open an incident of its own. It is silenced along with its alert. The link uses the portal’s address: set queryApi.publicUrl, or it is taken from your SSO redirect URL.

Shadow mode

Before a runbook is ever allowed to fix things by itself, shadow mode shows what it would have done. Set it to Propose, and shadow (autonomy: shadow). It proposes its fix exactly as Propose for approval does, so people stay in charge, and for each match from an alert or SLO it also records:

• What it would have done on its own — run the fix at once, or hold off because a guardrail said so (below).
• What happened instead, followed until the runbook’s verify.within has passed (longer while its proposal still waits):
  • A person ran the same fix, how much later, and whether it worked — the strongest evidence.
  • A person turned it down (with their reason) or fixed it another way.
  • It recovered with no fix: acting would have been unnecessary.
  • Nobody acted and it stayed broken: the case acting by itself is for.

The record shows on the proposal, on the investigation, and on the runbook, which sums up the last 30 days: how often it matched, would have acted and was held, how often people agreed with it and the fix worked, and how much sooner it would typically have acted. An investigation a person started isn’t counted: they were there to act.

Guardrails

The same checks stop a runbook that acts on its own; in shadow they only say what they’d have decided. The first that applies is the reason given.

Guardrail	Holds when
Switched off	Automatic fixes are switched off (the switch on the Runbooks page; admins).
Freeze window	It’s inside a freeze window: weekly times you set, such as a release window or peak trading, in any time zone.
Failed fix	The same fix didn’t work on that target within the hour.
Cooldown	Anyone acted on the target more recently than the runbook’s cooldown (15 minutes unless it says otherwise).
Rate limit	It would already have acted on that target max_per_hour times in the hour (2 unless it says otherwise, up to 12).
Blast radius	Deleting a pod with fewer than two others serving beside it, or draining or cordoning a node while another is already cordoned or fewer than two others could take its pods.

spec:
  autonomy: shadow
  cooldown: 15m        # at least this long between actions on one target
  max_per_hour: 2      # and at most this many an hour

Automatic fixes

Once a runbook’s shadow record shows it would have done what your team did, and sooner, let it act: set it to Act on its own (autonomy: automatic). When an alert or SLO starts firing and its investigation matches, the runbook runs the fix itself, unless a guardrail holds it. Then:

• The fix shows on the Fixes page and the investigation as Ran on its own, and the alert’s channels get a notice saying so.
• It is checked like any other. If it didn’t work, the alert’s channels hear about it, and the runbook won’t act on that target again for an hour.
• Held — by a guardrail, a site that doesn’t allow it, or a person already deciding on the same fix — it’s proposed for a person to approve instead, and the proposal says what held it.

An investigation a person started never acts by itself: they’re there to decide. The runbook’s record shows how often it acted, and how often that worked.

Who it acts as

An automatic fix runs as a dedicated Kubernetes service account at the site, <release>-autofix — never as a person, since nobody is there, and never as the core. The site’s actuator acts as that account, so the cluster’s audit log shows it. The account gets exactly the rights the allowed actions need, in the namespaces you allow, and nothing else. No cloud IAM is involved, so it works the same on every cloud and with no internet access.

Each site decides for itself. It’s off until the site’s own chart values allow it, and the site’s actuator refuses any automatic fix those values don’t allow, whatever the core asks:

# The site's chart (ultrvlt-site): what runbooks may do here on their own.
actuator:
  enabled: true
  automatic:
    enabled: true
    actions: [rollback, delete_pod]   # restart, scale, rollback, delete_pod, cordon, uncordon, drain
    namespaces: [shop, checkout]      # ["*"] for all; node actions need only the action

To manage the account and its permissions yourself, through GitOps or a policy engine, set automatic.serviceAccount.create: false with its name, and automatic.rbac.create: false. The chart then grants only the actuator’s right to act as that account. The Sites page lists what each site allows.

On the core’s own cluster, set the same under queryApi.actions.actuator.automatic in the core chart. If the core’s gateway requires enrolled client certificates, set it on the local connector’s site chart instead.

Turning it off

• Everywhere, from the portal: the automatic fixes switch on the Runbooks page (admins).
• Everywhere, from the deployment: queryApi.actions.automatic: false in the core chart. Runbooks set to act on their own then only propose.
• At one site: actuator.automatic.enabled: false in its chart, or delete the autofix account’s RoleBinding. This takes effect at once and doesn’t depend on the core.

Your own runbooks

Click New runbook, or keep runbooks in git with configuration as code (kind Runbook):

apiVersion: ultrvlt.io/v1
kind: Runbook
metadata:
  slug: restart-after-bad-deploy
spec:
  name: Restart a workload after a deploy broke it
  enabled: true
  autonomy: suggest          # suggest | approve | shadow | off (record only)
  approvals: 1               # approve, shadow: people who must approve (1 or 2)
  expires: 30m               # approve, shadow: how long a proposal waits
  cooldown: 15m              # shadow: least time between its actions on one target
  max_per_hour: 2            # shadow: most actions an hour on one target
  scope:
    namespaces: [shop]       # empty: everywhere
  when:
    finding: deploy          # deploy | replicas | node | pod
    of: any                  # deploy: focus (the service with the problem) | dependency | any
    within: 30m              # deploy: at most this long before the problem started
    vs_baseline: 3           # errors or p95 at least 3× the service's baseline
    min_error_rate: 0.01     # errors count from 1%
    min_latency_ms: 50       # latency counts when p95 rose at least this much
    min_requests: 100
  action:
    type: restart            # rollback | restart (deploy), scale (replicas)
  verify:
    signal: back_to_baseline
    within: 10m

• when.finding is the kind of finding the runbook starts from, and the action changes the workload that finding names.
• replicas findings take scaled_down: true (fewer replicas than before the problem) and the scale action with replicas: previous.
• pod findings take the same thresholds, compared with the pod’s sibling pods instead of a baseline, and the delete_pod action. Telling pods apart needs telemetry that says which pod it came from: the sensor’s does, and so do apps’ own spans sent to the gateway or a site’s connector (see Pod identity).
• node findings take pressure: true (short of memory, disk or process IDs, and still ready) or not_ready: true (not ready now), and the drain or cordon action. A node matches only when a workload in the problem runs on it, and it is in scope when one of its workloads is.
• overrides sets thresholds for particular services, for example overrides: {checkout: {vs_baseline: 5}}.

In the Helm chart

To manage runbooks with the deployment instead, list them under queryApi.runbooks. The portal shows them read-only. Use a built-in’s slug to change it:

queryApi:
  runbooks:
    - slug: rollback-bad-deploy
      enabled: true
      autonomy: approve
      approvals: 2
      scope: {namespaces: [shop, payments]}
      overrides: {checkout: {vs_baseline: 5}}

Removing a runbook from the list removes it; a built-in goes back to its defaults.

Records

Every match is recorded with its investigation, including those of runbooks set to Record only, and kept for 180 days. Proposed fixes and what became of them are kept for a year, like the actions they ran and the checks of whether they worked. The Runbooks page lists each runbook’s recent matches. Changes to runbooks are in the audit trail under Configuration.