Synthetic monitors

Synthetic monitors check an endpoint on a schedule, whether or not anyone is using it. They catch a broken or slow endpoint before a user does, and give quiet services — an admin API, an overnight job’s health check — a steady signal to watch.

Checks run from your own environment, not from ours: the core runs them from inside your cluster, so internal endpoints are just as checkable as public ones, and nothing leaves your network to do it.

Creating a monitor

Open APM → Synthetics and choose New monitor:

• Method and URL — any http:// or https:// endpoint the cluster can reach, internal or public.
• Run every — 30 seconds to 1 hour.
• Service (optional) — the service this endpoint belongs to. The monitor then appears on that service’s page, under Synthetics.
• Headers and body — anything the request needs. For an API key, turn on Secret and enter a key name instead of the value; see Secrets in headers.
• Assertions — what counts as passing. With none, any 2xx or 3xx response passes.

Assertion	Example
Status code	equals 200; one of 200,204
Response time	less than 500 ms
Body contains / matches	contains "ok"; matches ^\{"status"
JSON value	data.status equals up; items[0].count greater than 0
Response header	content-type contains json
Days until certificate expires	greater than 14

A check fails on the first assertion that doesn’t hold, and the result says which one and why — status 503, timed out, data.status is degraded.

Other kinds of check

Choose the kind when you create a monitor:

Kind	What it does	Passes when
HTTP	Requests a URL	The assertions hold (any 2xx/3xx with none)
Multi-step API	A sequence of requests in one trace	Every step’s assertions hold
TCP	Connects to a host and port; optionally a TLS handshake, a line to send and a reply to expect	It connects (and the reply contains what you expect)
DNS	Resolves a name as A, AAAA, CNAME, MX, TXT, NS or SRV, optionally against a specific DNS server	The name resolves and the answer assertions hold
gRPC health	Calls the standard gRPC health check, optionally over TLS	The service answers SERVING
Push	Your own check reports to Ultraviolet — see below	A passing result arrives in time

TCP and gRPC checks can warn about expiring certificates like HTTP ones; every kind can assert on response time. Response time includes looking up the name, so when a run fails on response time its reason says how much of it was the lookup — for example response time (ms) is 230 (212 ms of it looking up the name).

Kubernetes service names ending in .cluster.local are looked up as complete names, without trying the cluster’s search domains first, so in-cluster checks measure the service rather than the DNS search path. To do the same for another complete name, end it with a dot (db.internal.example.com.).

Multi-step API checks

A multi-step check runs requests in order, the way a client uses an API: log in, then fetch the profile with the token, then read the cart. Each step can save a value from its response as a variable — a JSON value (data.token), a header (Location) or the first capture group of a regex — and later steps use it as ${name} in their URL, headers or body. You can also set starting variables, such as a test user’s name.

The run stops at the first failing step, and the monitor’s page shows each step’s outcome and timing. All of a run’s requests share one trace, so a failed step opens the backend trace for that whole sequence.

Reading the results

The Synthetics list shows each monitor’s status, uptime over the selected window, p95 response time and when it last ran. No recent results means the monitor hasn’t reported for three intervals — usually that the checker isn’t running.

A monitor’s page shows:

• Uptime and latency per location, and how many days its TLS certificate has left.
• Response time by phase — DNS lookup, connecting, TLS handshake, waiting for the first byte, and downloading — so you can tell a slow network from a slow server.
• Recent runs, with the failed assertion and the start of the response body for each failure.

From a failed check to the trace

Every check sends a trace header with its request, and Trace on a run opens what Ultraviolet saw of it: the check itself and the request as it crossed the network to the service. Where the service passes the trace header on to the services it calls — any service instrumented with OpenTelemetry does, as do most service meshes — their spans join the same trace, so a failed check opens the whole request: which service was slow, which query failed. Without that, the trace shows the request to the first service and its response.

Locations

A monitor runs from every location it lists, and its page shows uptime and latency for each — so you can tell “down everywhere” from “unreachable from one office”.

• core — the cluster Ultraviolet runs in.

• A site — any remote site whose connector has prober.enabled=true runs the monitors that list the site’s name. Checks run from the site’s own network, so internal endpoints there are checkable too, and results travel back through the site’s connector with the rest of its telemetry.

• Outside your network — to see a public endpoint the way your users do, run a standalone location on a small machine anywhere: a connector and a checker together, enrolled like a site. Add a site in Sites → Add site named after the location (for example outside-eu), then on the machine use the compose file from the Ultraviolet repository (deploy/standalone-prober/docker-compose.yml) with the join token and addresses the wizard shows:

  docker compose up -d

  Then add outside-eu to the monitors that should run there.

A location’s results are recorded under the identity it enrolled with, so a location can’t report results as another. With several locations, alerting can wait until a monitor fails at more than one (At this many locations), so a single network path’s trouble doesn’t page anyone.

Alerting

Turn on Alert when it fails in a monitor’s settings:

• Failed runs in a row — how many consecutive failures at a location make it failing. Two is a good default: one blip doesn’t page anyone.
• At this many locations — for a monitor that runs from several places, how many must be failing before it alerts.
• Certificate warning — alert when an HTTPS certificate expires within this many days (0 turns it off).
• Send to — which notification channels hear about it; empty sends to all of them.

A missing result counts as a failure. If a monitor stops reporting — its checker isn’t running, or a push monitor’s job didn’t run — it’s treated as failing once a result is overdue: three intervals for a built-in check, the expected interval plus its grace period for a push monitor.

You get one notification when the monitor goes down and one when it recovers, with how long it was down. A monitor’s page lists its alert history.

Push monitors: your own checks

Anything you can script can be a monitor — a browser test in Playwright or Cypress, a multi-step API flow, a backup job, a nightly data pipeline. It runs wherever you like and reports each run to Ultraviolet, where it gets the same uptime, alerting and SLOs as a built-in check.

1. On Synthetics, choose New monitor → Push, and say how often a result should arrive and how late it may be.
2. On the monitor’s page, choose Issue token. Copy it — it’s shown once. Rotate token issues a new one and stops the old one.
3. Have your check report each run to the ingest gateway’s HTTP address:

curl -X POST https://gateway.example.com/v1/synthetics/results \
  -H "Authorization: Bearer $UV_PUSH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"schema": 1, "passed": true, "duration_ms": 1830, "location": "ci",
       "steps": [{"name": "log in", "duration_ms": 640, "passed": true},
                 {"name": "check out", "duration_ms": 1190, "passed": true}]}'

For a job that only needs to say “I ran”, send a heartbeat instead — ?status=fail reports a failure:

curl -fsS -X POST https://gateway.example.com/v1/synthetics/<monitor-id>/ping \
  -H "Authorization: Bearer $UV_PUSH_TOKEN"

A missed heartbeat is how you find out a cron job stopped running.

The results format

Field	Required	Description
schema		1.
passed	yes	Whether the check passed.
duration_ms		How long the run took.
timestamp		When it started (RFC 3339); defaults to now. Must be within the last day.
location		Where it ran, e.g. ci or eu-laptop; defaults to push.
error		Why it failed.
status_code		An HTTP status, if there was one.
run_id		A unique id per run. A repeat within 15 minutes — a retried request — is ignored (by each ingest replica, if you run several).
trace_id		The run’s W3C trace id (32 hex digits). If your check sent a traceparent, the run links to the backend trace.
steps		[{"name", "duration_ms", "passed", "error"}] — shown step by step on the monitor’s page.
monitor_id		Optional; the token already identifies the monitor.

Send one result, or up to 100 as {"results": [...]}. Each monitor accepts up to 5 reports a second.

Push monitors need the gateway’s HTTP listener (ingestGateway.otlpHttp.enabled), and for checks outside the cluster, a way to reach it — see Configuration. With remote sites enabled, that listener uses HTTPS with Ultraviolet’s own certificate authority: the token is still all a push needs, but your client must trust that CA — download it from the enrollment endpoint’s /ca and pass it along (for curl, --cacert ca.pem), using the gateway name your Sites page shows.

Note

A pushed result records what your check reports. Ultraviolet can vouch for the checks it runs itself; for pushed ones, the token tells you which monitor a result is for, not that the check was right.

Service level objectives

A monitor can back an SLO — useful for a service with too little real traffic to measure. On the monitor’s page choose Use for an SLO, or on SLOs create one with the source Synthetic monitor: availability counts passing runs, latency counts runs faster than the threshold.

Secrets in headers

API keys and tokens are never stored in a monitor. Instead, put them in a Kubernetes Secret in Ultraviolet’s namespace and name it in the chart:

kubectl -n ultrvlt create secret generic synthetics-secrets \
  --from-literal=shop-api-key='…'

prober:
  secretsSecret: synthetics-secrets

Then, in the monitor’s header, turn on Secret and enter shop-api-key.

What checks can reach

The checker can reach anything the cluster can, which is the point — and also why it’s guarded:

• Always refused: the cloud metadata endpoint (169.254.169.254) and other link-local addresses, loopback, and multicast. A URL whose name resolves to one of these is refused too, including after a redirect.
• Allowed: private and public addresses.
• Only editors can create or change monitors, and every change is recorded in the audit log.

To narrow it further, block ranges with prober.egress.denyCIDRs, or turn on prober.networkPolicy to restrict the checker’s network access to the destinations you list. See Synthetic monitors in the configuration reference.