Skip to content

Configuration as code

Everything your team builds in the portal — dashboards, alert rules, notification channels, SLOs, synthetic monitors, log-pipeline rules and cost budgets — can live as YAML files. Keep them in git, review changes like code, promote them from a test core to production, and see when someone has changed something by hand.

Terminal window
ultrvlt export -o config/ # write every definition as files
ultrvlt diff -f config/ # what would change (exits 2 if anything)
ultrvlt apply -f config/ # make the core match the files

It needs two settings:

  • ULTRVLT_URL — the core’s address, for example https://observability.example.com.
  • ULTRVLT_TOKEN — an API token. diff and export work with any role; apply needs editor.

The command ships inside the core’s query-api image, so a CI job needs nothing else:

Terminal window
docker run --rm -v "$PWD/config:/config" \
-e ULTRVLT_URL -e ULTRVLT_TOKEN \
--entrypoint /ultrvlt ghcr.io/ultrvlt/query-api:<version> diff -f /config

export writes one file per definition, in a folder per kind:

config/
alert-rules/checkout-errors.yaml
dashboards/checkout-overview.yaml
notification-channels/oncall-webhook.yaml
...

Each file is one document:

apiVersion: ultrvlt.io/v1
kind: AlertRule
metadata:
slug: checkout-errors
spec:
name: Checkout error rate
metric: error_rate
service: checkout
comparator: gt
threshold: 0.05
window_secs: 300
notify_channels: [On-call webhook]
enabled: true
  • kind is one of Dashboard, AlertRule, NotificationChannel, SLO, SyntheticMonitor, LogPipelineRule and CostBudget.
  • slug is the definition’s lasting identity: re-applying a file updates the same definition, on any core. A definition made in the portal gets its slug from its name the first time it is exported, and keeps it even if it’s renamed later.
  • spec holds the same fields as the portal form. A field you leave out takes its default, so files stay short.

Files can be organised however you like: apply reads every .yaml, .yml and .json file under the paths you give it, and a file can hold several documents separated by ---.

Alert rules, SLOs, synthetic monitors and budgets name their notification channels by name, as in the portal. No secrets are ever in these files: channels and synthetic monitors refer to secrets by key, and the values stay in your deployment’s secret store.

apply checks every document first — a misspelled field, an unknown kind, an invalid value or a duplicate slug stops it before anything changes — then creates and updates definitions, notification channels first. Each change goes through the same checks as the portal and is recorded in the audit trail under the token’s name, with one entry for the whole apply.

If a change is refused part-way (for example, an alert rule names a channel that doesn’t exist), apply stops there and reports which changes were made; nothing after the failure is attempted. Fix the file and run it again: what was already applied shows as unchanged.

A definition made in the portal that has never been exported is matched to a file whose slug comes from the same name, so applying a file for it updates it rather than creating a copy.

By default apply only creates and updates: definitions that aren’t in your files are left alone, so you can bring definitions under git a few at a time.

apply --prune also deletes definitions that your files don’t name — but only of the kinds your files contain. A directory with only dashboards in it can never delete an alert rule. Run diff --prune first to see exactly what would go.

A typical setup:

  • On a pull request, run ultrvlt diff -f config/ with a viewer token and show the output in the review. Exit code 2 means the files would change the core.
  • On merge, run ultrvlt apply -f config/ with an editor token.
  • On a schedule, run diff to catch changes made by hand in the portal.

Definitions are part of the core’s database, so your backups include them and a restore brings them back. An export in git is a second, portable copy: it can be applied to a fresh core, or to a different one.