LLM observability

APM → LLM calls shows every call your services make to a language model: which model, how many tokens went in and out, how long it took, how often it failed or was rate-limited, and what it cost. It works for hosted models such as OpenAI, Anthropic and Gemini, and for models you run yourself.

Getting data in

The calls come from your services’ LLM instrumentation, sent like any other traces to your site’s connector or the core. Any of these work:

• OpenTelemetry GenAI instrumentation: for example opentelemetry-instrumentation-openai-v2 for the OpenAI Python SDK, or the instrumentation for your language and model provider.
• OpenLLMetry (the Traceloop SDK), which also covers LangChain, LlamaIndex and many providers.
• OpenInference (Arize), including its LangChain and LlamaIndex integrations.

Calls are read from whichever of these a service uses. Frameworks often wrap each model call in workflow, agent or tool steps; only the model calls themselves are counted, so nothing is counted twice.

Calls seen without instrumentation

Calls to model APIs also appear under Seen on the network with no instrumentation at all: which service called which provider, how often, how fast, how many failed, and how many were rate-limited. The page recognises:

• Hosted providers: OpenAI, Azure OpenAI, Anthropic, the Gemini API, Vertex AI, Amazon Bedrock, Mistral, Cohere, Groq, DeepSeek, xAI, Together AI and OpenRouter.
• Your own model servers and proxies: any server offering the OpenAI-compatible API (vLLM, LiteLLM, LM Studio, and others), Ollama, and Text Generation Inference.

Quota left is the smallest share of a provider’s rate limit that remained, read from the limits the provider reports with each response (OpenAI-style APIs and Anthropic). A low figure warns you before calls start being rate-limited.

These calls have no token counts or cost. The model is shown only when the request’s address names it (Gemini, Vertex AI, Bedrock and Azure OpenAI deployments); add instrumentation to see tokens and cost for every call.

Today this covers Python services, including the official OpenAI, Anthropic and Google SDKs, and unencrypted calls to model servers in your clusters. Node.js, Go and Java services aren’t recognised yet: instrument them instead.

Model servers and GPUs

If you run models yourself, the page also shows your inference servers and the GPUs under them, from the metrics they already publish:

• Model servers: vLLM and Text Generation Inference. For each server and model: requests running and queued, how full vLLM’s KV cache is, requests completed, input and output tokens, time to the first token (vLLM) and request time. Queued requests and a nearly full KV cache are the first signs a server needs more capacity.
• GPUs: from NVIDIA’s DCGM exporter. For each GPU: its model, the pod using it, how busy it was over the period, memory used, peak temperature and power. A GPU that stays near 0% is one you’re paying for and not using.

Send these metrics in the way you send other Prometheus metrics: scrape the servers’ /metrics endpoints and the DCGM exporter with Prometheus and remote-write them, or scrape them with an OpenTelemetry Collector and send them over OTLP. Turn on the DCGM exporter’s Kubernetes mapping to see which pod is using each GPU. Ollama doesn’t publish metrics; calls to it appear under Seen on the network.

Prompts are never stored

These libraries record prompt and response text by default. Ultraviolet removes it as it arrives: span attributes holding prompts, responses, tool arguments or results, and the message events that carry the conversation. Only metadata is kept (model, tokens, timing, errors). There is nothing sensitive to leak and nothing to redact later.

What you see

• Totals for the period: calls, error rate, rate-limited calls (HTTP 429), input and output tokens, and cost.
• Cost and tokens over time.
• Breakdowns by model and by service, with p95 latency per model.

LLM spend also appears at the top of Cost, next to what your clusters cost.

Prices

Cost uses a table of prices per million tokens, which works offline. It starts with list prices for common hosted models. Click Prices to see it, and, as an administrator, to edit it to match what you actually pay.

• A model is priced by the longest entry its name starts with, so gpt-4o-mini covers gpt-4o-mini-2024-07-18.
• A model with no entry, such as one you host yourself, counts as free, and the page lists it so you can add a price (for example your GPU cost per token).
• Reset to defaults returns to the shipped list.