Connecting remote sites securely

You can point one core at many locations: another cluster, another region, or a cluster you treat as less trusted than the one running the core. Each site runs a small connector that forwards its telemetry to the core. This page is the operator how-to for doing that securely — the network setup, the trust model, and how to keep the remote up to date.

For the product-level tour of the Sites view, see Multi-site.

The trust model

The rule that shapes everything below: the site authenticates to the core, and the core never trusts the network.

• The connector presents a client certificate issued by your core’s own enrollment authority. The core verifies it and derives the site’s identity (which site, which cluster) from the certificate — never from what the sender claims. A site can’t assert that it is some other site, so a less-trusted location can never write into a trusted site’s data.
• The connection is outbound-only: the site dials the core. You never open an inbound path to the remote location.
• The only thing the core ever exposes to a site is the ingest endpoint. The portal, query API, and database are never reachable from a site.
• If the link drops, the connector buffers to disk and backfills on reconnect, so a network blip loses nothing. The same buffer holds a site’s telemetry while the core asks it to slow down (an ingest quota); a batch the core can never accept is set aside and logged, so it can’t hold up the rest.

Because of this, the entire surface a site can reach on your core is “authenticated telemetry to one port, and nothing else.”

Caution

Authentication is the certificate, not a network allowlist. An IP allowlist or security-group rule is valuable defense-in-depth — it shrinks who can reach the endpoint — but it is not what proves identity. Always enrol sites so the connection is mutually authenticated; don’t rely on network reachability alone.

Prepare the core

Three one-time settings on the core turn on remote-site fan-in. All are Helm values on the core (umbrella) chart — see the Configuration reference for the full table.

1. Enable enrollment. Set enrollment.enabled=true. This stands up the core’s enrollment authority (so it can issue join tokens and site certificates), turns on the Add site action in the portal, and switches the ingest endpoint to mutual TLS.

   Caution

   Switching the ingest endpoint to mutual TLS means it no longer accepts telemetry in the clear — including from a sensor or collector running on the core’s own cluster. If your core collects its own telemetry, see If the core also collects its own telemetry before you enable enrollment.

2. Publish the ingest endpoint. Set ingestGateway.wan.enabled=true and scope it to your sites’ egress addresses. This is the one endpoint a site reaches — see Expose the ingest endpoint for the topology that matches your network.

3. Expose the enrollment endpoint so a site can exchange its join token for a certificate, and record the public addresses so the wizard can hand sites a ready-to-run command — see Expose the enrollment endpoint.

You’ll also need admin access to the core (to run the Add site wizard) and cluster access to the remote location (to install the connector).

Expose the ingest endpoint

The connector needs a stable address for the core’s ingest endpoint. The core’s ingest service is internal by default, so you publish it with a load balancer scoped to exactly the traffic you intend to allow. Because the load balancer operates at the connection level, the core still terminates the encrypted, mutually-authenticated session itself — the load balancer only forwards it.

The chart publishes this for you: set ingestGateway.wan.enabled=true and supply your provider’s load-balancer annotations and the source ranges to allow. The two common topologies below are just different values for that one setting. (It must stay a connection-level balancer — don’t put the ingest endpoint behind an application load balancer that terminates TLS, or the certificate-based identity the core relies on is lost.)

Topology A — over the internet (or across accounts)

For a site in another network or another account:

• Publish the ingest endpoint on a dedicated, internet-facing load balancer for the ingest port only. Keep it separate from the portal — never put the ingest endpoint behind the same public entry point as the UI.
• Restrict source addresses to the egress ranges your remote locations actually use. This is defense-in-depth on top of mutual authentication.
• Prefer a private link over the public internet when both ends are in the same cloud. On AWS, publish the internal load balancer as a VPC endpoint service (PrivateLink) and have the consumer account attach a private endpoint — the traffic never traverses the public internet, and you expose nothing to the world.

On AWS, these core-chart values front the ingest endpoint with an internet-facing network load balancer, restricted to known source ranges:

# values for the core (umbrella) chart
ingestGateway:
  wan:
    enabled: true
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    # Only the egress ranges your sites connect from — not 0.0.0.0/0.
    sourceRanges:
      - 203.0.113.0/24

Topology B — a less-trusted cluster on the same network

This is the case where a cluster shares your network (for example, the same VPC) but runs workloads you don’t fully trust, and you want only Ultraviolet traffic to be able to cross into the cluster running the core.

Two layers give you that guarantee:

1. An internal load balancer whose security group admits the ingest port from only the source cluster’s security group — not the whole network range, and never the public internet. That single scoped rule is the “only Ultraviolet traffic crosses” guarantee: the sole path from the less-trusted cluster into the core’s cluster is a connection to the ingest port. Everything else stays denied, because nothing else admits that source.
2. A network policy on the core side so that only the ingest gateway pods accept the connection, and the namespace can’t be used as a stepping stone.

# values for the core (umbrella) chart
ingestGateway:
  wan:
    enabled: true
    annotations:
      service.beta.kubernetes.io/aws-load-balancer-type: external
      service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
      # Internal only — never internet-facing for this case.
      service.beta.kubernetes.io/aws-load-balancer-scheme: internal
      # Attach a security group that admits the ingest port from ONLY the source
      # cluster's node/pod security group (configure the rule on that SG).
      service.beta.kubernetes.io/aws-load-balancer-security-groups: sg-ultrvlt-ingest

A matching network policy that lets only the ingest gateway receive traffic (the chart doesn’t create this — add it alongside your install):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ultrvlt-ingest-only
  namespace: ultrvlt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: ultrvlt
      app.kubernetes.io/component: ingest-gateway
  policyTypes: [Ingress]
  ingress:
    - ports:
        - port: 4317
          protocol: TCP

Note

Even on a shared internal network, a less-trusted site must still be enrolled and mutually authenticated — don’t send its telemetry in the clear. Scoping the network is what stops reaching the core; the certificate is what stops impersonating a trusted site. You need both.

Expose the enrollment endpoint

A site enrols once, on first start: it exchanges its one-time join token for its own certificate. So the site needs to reach the core’s enrollment endpoint too — briefly, and only to enrol and later renew. Publish it over HTTPS, scoped to the same source ranges as the ingest endpoint.

An application load balancer is the right fit here (unlike ingest, this is plain HTTPS). On AWS, an Ingress like this fronts the enrollment service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ultrvlt-enroll
  namespace: ultrvlt
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: <your-acm-cert-arn>
    alb.ingress.kubernetes.io/healthcheck-path: /healthz
    # Same site egress ranges you allowed on the ingest endpoint.
    alb.ingress.kubernetes.io/inbound-cidrs: 203.0.113.0/24
spec:
  ingressClassName: alb
  rules:
    - host: enroll.ultrvlt.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ultrvlt-enrollment
                port:
                  number: 8090

Then tell the core the public addresses sites should dial, so the Add site wizard can hand each site a ready-to-run command (rather than making you assemble it):

# values for the core (umbrella) chart
queryApi:
  enroll:
    publicUrl: https://enroll.ultrvlt.example.com
    corePublicEndpoint: ingest.ultrvlt.example.com:4317
    coreServerName: ingest-gateway

coreServerName is the name a site verifies on the core’s certificate; leave it at the default unless you changed ingestGateway.mtls.serverNames.

If the core also collects its own telemetry

Enrollment puts the core’s ingest endpoint into mutual-TLS mode, so it stops accepting telemetry in the clear — including from a sensor or collector running on the core’s own cluster. Give those local producers a connector of their own to authenticate through: install the same ultrvlt-site chart on the core, with its bundled producers off, pointing at the in-cluster ingest and enrollment endpoints. Enrol it exactly like a remote site (mint it a join token from the wizard) — it just happens to be co-located, so it dials service names instead of public addresses:

helm install ultrvlt-site oci://ghcr.io/ultrvlt/charts/ultrvlt-site \
  --namespace ultrvlt \
  --set aggregator.coreEndpoint=<core-release>-ingest-gateway:4317 \
  --set aggregator.coreServerName=ingest-gateway \
  --set aggregator.enrollUrl=http://<core-release>-enrollment:8090 \
  --set aggregator.joinToken.value=<join-token-from-the-wizard>

Then repoint the core’s own data sources at this connector’s Service by setting their endpoint overrides on the core chart — for example sensor.otlpEndpoint, clusterState.otlpEndpoint, and logCollector.otlpEndpoint to the connector’s Service — ultrvlt-site-aggregator:4317 for a release named ultrvlt-site as above (<release>-ultrvlt-site-aggregator:4317 for any other release name; kubectl get svc shows it). The core’s telemetry then flows through the connector and is recorded under its own site identity, alongside your remote sites.

The same goes for the core’s synthetic monitor checker: set both prober.otlpEndpoint and prober.assignmentsEndpoint to the connector’s Service. Its checks then run as that site — list the site’s name, rather than core, on monitors that should run from the core.

Connect the site

With the endpoint reachable, add the site from the core:

1. In the portal, choose Add site (an admin action). Name the site and cluster and choose how long the one-time join token stays valid.
2. The wizard hands you a ready-to-run command that starts the connector at the remote location, carrying the join token and the address of your ingest endpoint. It’s a single-container docker run — the quickest way to bring a site online.
3. Run that command at the remote location. The connector enrols — exchanging the one-time token for its own site certificate — and begins forwarding.
4. Point your data sources at the connector: enable the sensor on the remote cluster (and log/infrastructure collection if you want them) so their telemetry flows through the connector to the core.
5. The wizard confirms as soon as the site’s first data arrives. The site then appears on the Sites page, tagged with its identity, and its data joins the single cross-site view.

Running the connector on Kubernetes

For a remote Kubernetes cluster, install the connector as its own small Helm release instead of the docker run command — you get a durable volume for the buffer and site identity, a Service your in-cluster sensors can target, and it can run the sensor and collectors for you. It’s a separate, lightweight chart (ultrvlt-site) — not the full core chart:

helm install ultrvlt-site oci://ghcr.io/ultrvlt/charts/ultrvlt-site \
  --namespace ultrvlt --create-namespace \
  --set aggregator.coreEndpoint=ingest.ultrvlt.example.com:4317 \
  --set aggregator.coreServerName=ingest-gateway \
  --set aggregator.enrollUrl=https://enroll.ultrvlt.example.com \
  --set aggregator.joinToken.value=<join-token-from-the-wizard> \
  --set sensor.enabled=true \
  --set clusterState.enabled=true

• coreEndpoint and enrollUrl are the public ingest and enrollment addresses you published above.
• coreServerName is the name the connector verifies on the core’s certificate — the core’s ingestGateway.mtls.serverNames (default ingest-gateway), not the ingest host name.
• The bundled producers (sensor, clusterState, logCollector) are off by default and each exports to the connector automatically — turn on the ones this cluster should feed the core.
• prober.enabled=true runs synthetic monitors from this site: any monitor that lists the site’s name. To wire your own producers instead, point them at the connector’s in-cluster Service: ultrvlt-site-aggregator:4317 for a release named ultrvlt-site, else <release>-ultrvlt-site-aggregator:4317.

Tip

On a busy cluster the sensor can need more memory than its default limit. If sensor pods restart with an out-of-memory status, raise sensor.resources.limits.memory (for example to 1Gi).

Keeping a remote site up to date

A Kubernetes remote site runs the standalone ultrvlt-site connector chart, so you upgrade it the way you installed it — run an upgrade against the remote cluster, reusing that site’s values:

helm upgrade ultrvlt-site oci://ghcr.io/ultrvlt/charts/ultrvlt-site \
  --version <version> -n ultrvlt --reset-then-reuse-values

(If you started the connector with the wizard’s docker run command instead, upgrade it by pulling the newer image and restarting the container — its volume carries the buffer and identity across the restart.)

Upgrade a site the same way you’d upgrade any workload in that cluster: on the cadence and during the maintenance window that location’s owners prefer. The connector’s on-disk buffer covers the brief restart, so an upgrade doesn’t lose data.

Tip

The Sites page shows when each site was last seen, so after an upgrade you can confirm the site reconnected and is reporting again.

Decommissioning a site or revoking a certificate

When a site closes, its cluster is handed to someone else, or you suspect its certificate has leaked, stop trusting it from the core. On the Sites page, Enrolled sites (admins only) lists every enrolled site and the certificates issued to it.

• Decommission revokes every certificate issued to the site. Within a minute the core refuses its data — on connections that are already open, not just new ones — and the site can no longer renew its certificate. Join tokens minted for the site before you decommissioned it stop working too.
• Revoke on a single certificate (expand the site’s row) refuses just that one, for example when one connector’s key may have leaked, and leaves the site’s other connectors running.

A decommissioned site’s connector keeps buffering to disk and retrying, so nothing it collected is lost while you decide. If you decommissioned a site by mistake, Undo trusts its certificates again, and the buffered data flows in. To bring a decommissioned site back for good, enrol it again with a new join token from Add site.

Both actions are recorded in the audit log. The core keeps enforcing them even if its enrollment service is briefly unavailable.