Sizing and capacity

Three things decide how big an install needs to be:

1. How many nodes you monitor — each runs one sensor, whose cost grows with the traffic on that node.
2. How much telemetry arrives per second — spans, log lines and metric points. This drives the core’s CPU.
3. How long you keep it — this drives disk.

Start from the numbers below, then confirm on the Core status page (under System): it shows disk used and free, how well your data compresses, and how fast each site’s telemetry is arriving. Grow the volume before it passes 80% full — Core status turns amber there and red at 90%.

Where these numbers come from

Measured on our public demo — a two-node Kubernetes cluster (4 vCPU / 16 GB each) running a 12-service online shop under real load, from about 50 to 215 spans per second. Figures marked estimate extend those measurements to larger installs; validate them against Core status, and talk to us before sizing a large deployment.

Disk

Storage per day is, for each kind of telemetry:

records per second × stored bytes per record × 86,400

Telemetry	Stored size per record	Notes
Spans (traces)	~85 bytes	Compressed about 6.7× from ~560 bytes
Log lines	~ raw size ÷ 10	~45 bytes for a typical 500-byte line (message plus Kubernetes metadata); compressed about 11×
Metric points	~5 bytes	Compressed about 25×

Per day, at a steady rate:

Rate	Spans	Log lines (500 B)	Metric points
100 / s	0.7 GB	0.4 GB	0.04 GB
1,000 / s	7.3 GB	3.9 GB	0.4 GB
10,000 / s	73 GB	39 GB	3.9 GB

Multiply by your retention (30 days by default; change it on the Retention page), then provision about 1.5× that: the storage engine needs free space to merge data in the background, and you want headroom before the 80% mark.

Example: 1,000 spans, 1,000 log lines and 1,000 metric points per second ≈ 11.6 GB/day → 30 days ≈ 350 GB → provision about 520 GB. At 7 days’ retention, about 120 GB.

Set the size with clickhouse.persistence.size (default 50Gi). Telemetry volume is usually dominated by spans and logs; if logs dominate, the log pipeline can drop or sample noisy sources before they’re stored.

Sensors (per node)

	Measured	Chart default
Memory	~320 MiB, flat with traffic	128 MiB request, 512 MiB limit
CPU	~25 millicores idle, + ~0.6 millicores per span/s on that node	100 millicores request

So a node serving 100 spans per second uses about 85 millicores; estimate: a node serving 1,000 spans per second about 0.6 cores (we measured up to ~110 per node and the growth was linear). Keep the 512 MiB limit; raise it only if a sensor is restarted for running out of memory — most likely on nodes running hundreds of distinct programs, which the sensor inspects when it starts.

Sensors compress what they send (on by default), so their network traffic to the core is roughly 1 KB per 10 spans.

The core

Measured at up to ~600 records per second in total (spans, logs and metrics together):

Component	CPU	Memory
Storage (ClickHouse)	0.1–1 core, swinging with background merges and queries	~1.3 GiB
Ingest gateway	~15–30 millicores	~40 MiB
Query API	a few millicores at rest; grows with people using the portal	~20 MiB
Log collector (per node, if enabled)	~35–80 millicores	~75 MiB

Starting points by total ingest rate:

Total records / s	Storage (ClickHouse)	Ingest gateway	Query API
up to ~1,000	1–2 vCPU, 4 GiB — the chart default (1 vCPU requested, 4 GiB limit) is enough	defaults	defaults
up to ~10,000 (estimate)	4–8 vCPU, 16 GiB	500 millicores, 512 MiB	1 vCPU, 1 GiB
beyond	talk to us

Storage CPU depends more on how much people query — dashboards left open, wide log searches, alert and SLO rules — than on ingest itself. Size up when the portal feels slow, and turn on query protection (below) before you invite many users.

Set these with each component’s resources value — clickhouse.resources, ingestGateway.resources, queryApi.resources — see the configuration reference.

High availability

By default each core service runs as one pod. To keep the portal, the API and ingest up through a node failure or a node drain, run more than one:

queryApi:
  replicaCount: 2
ingestGateway:
  replicaCount: 2
portalUi:
  replicaCount: 2

Above one replica, each service’s pods prefer different nodes, and a disruption budget lets a drain take only one at a time. You need at least two nodes for the core to benefit, and each node must be able to run the whole core by itself: storage included. While one node is drained or lost, everything it ran moves to the others, and storage (requesting most of a small node) can’t start again until there is room for it.

Sensors and remote sites each keep one long-lived connection to an ingest gateway, so the load spreads across gateway replicas by connection rather than by request. If a gateway goes away, its senders reconnect to another one and resend anything that wasn’t acknowledged.

With queryApi.replicaCount above one, the API pods serve requests only, and a separate query-api-worker pod runs the background work: alert and SLO evaluation, usage metering, license call-home, cost jobs. The API pods read that work’s results and status from storage, so it doesn’t matter which pod answers. If the worker goes down, the portal stays up and evaluation pauses until Kubernetes restarts it, usually within a minute. It then resumes where it left off: alerts that were firing stay firing and aren’t re-notified.

A few limits apply per pod rather than to the whole core:

• queryApi.rateLimit and failed sign-in lockouts are counted by each API pod, so a user can make up to replicaCount times rps in total.
• Ingest quotas (ingestGateway.quota) and span sampling apply per gateway pod. Quotas are a backstop against a runaway site, so scale the rate down by the replica count if you need the total exact.
• Push monitor retries: a result resent with the same run_id is ignored only when it reaches the same gateway pod, so occasionally a retry is recorded twice.

Storage runs as a single instance by default. Its data sits on a persistent volume and survives a restart. While it is restarting or down, each ingest gateway buffers incoming telemetry on disk and writes it once storage is back (see Ingest buffer), so a storage upgrade or node failure delays data rather than losing it. Only the portal is unavailable until storage returns.

To keep the portal up through that too, turn on replicated storage: two or three storage replicas on separate nodes, each with all the data. When one goes down, the portal and ingest carry on with another, and the returning replica catches up on its own. Replication needs a coordination quorum (three small pods). If two of the three are lost, storage stays readable but stops taking writes, and the ingest gateways buffer incoming data until the quorum is back.

By default a single query may use as much of the storage engine as it likes, so one very wide search can slow ingest and everyone else’s queries. For a shared production install, set limits:

queryApi:
  queryLimits:
    maxExecutionTime: 60s         # stop any query after a minute
    maxMemoryUsage: 1073741824    # 1 GiB — about a quarter of the storage engine's memory limit
  rateLimit:
    rps: 20                       # sustained requests per second, per user
    burst: 120                    # a dashboard load fires many at once

A query that hits a limit fails with a clear error instead of degrading the whole system. Scale maxMemoryUsage with clickhouse.resources.limits.memory.

Ingest quotas

With several sites on one core, cap each so a runaway one can’t take the others’ capacity. Set the quota well above a site’s normal peak — two to three times what Core status shows for it — so it only bites on a real runaway:

ingestGateway:
  quota:
    recordsPerSecond: 3000      # default per site
    perSite:
      datacenter-east: 10000    # a bigger site

A throttled remote site buffers and catches up, so a quota delays data rather than losing it. See Ingest quotas.

Span sampling

Spans are usually the largest share of storage. If trace storage outgrows your disk budget, span sampling keeps one in N ordinary requests while keeping every error and slow request; request rates, latency, alerts and SLOs stay exact, from a per-minute summary of all traffic.

Only requests are sampled; connection records (one per TCP connection) are always kept. On our demo, requests are about two-thirds of stored spans, so keeping one in ten would cut span storage to roughly 40% — the kept requests, the errors and slow requests kept in full, and every connection record. Logs and metrics are unaffected. Check the split for your own traffic on Core status after turning it on.