A turn takes ten seconds. Where did the time go?
It might have waited for a worker, processed a long uncached input, streamed slowly, retried a failed request or spent most of its time in a tool. The same end-to-end number can describe very different problems.
I want a trace that lets me separate those explanations and a cost record that includes the attempts the user never saw.
Start with the units of work
A session contains turns. A turn can contain several logical model calls and tool executions. A logical call can have multiple physical attempts because of retries. These identities should remain related without being collapsed into one span.
The root turn span measures the experience of the request. Child spans identify model attempts, backoff and tools. Store session/turn/call/attempt identifiers on spans so we can follow one execution. Avoid putting every session ID into metric labels: an unbounded set of label values is expensive to aggregate.
Inspect one synthetic turn
A 2,800ms synthetic turn. Select a span to inspect what its timing means.
This is a synthetic waterfall. It demonstrates timing boundaries and unknown usage, not the speed of this runtime or any provider.
Which token latency?
For the client, useful timestamps include request start, first response bytes, first visible content, subsequent content arrivals, completion and cancellation. First response bytes may be headers or non-content events. First visible content may follow reasoning or tool-use events. State which boundary a reported time-to-first-token actually uses.
A streaming network chunk may contain several tokens, part of a token’s text, or metadata. Without token-level events or an appropriate mapping, call the intervals inter-content or inter-chunk gaps. Do not silently relabel them as server token-generation times.
Server-side queueing and kernel timings require provider or self-hosted instrumentation. A client trace sees their effects but cannot uniquely decompose them. Comparing two clocks also requires synchronization or a protocol that provides meaningful durations.
What should the dashboard answer?
| Question | Measurement |
|---|---|
| How long did the user wait? | End-to-end turn duration and the chosen first-content boundary |
| Did the stream stall? | Content-arrival gap distribution, including the measurement point |
| Was a tool responsible? | Tool duration, outcome and returned-output size |
| Did a retry hide work? | Physical attempt count, backoff and attempt outcomes |
| Was prior context reused? | Raw usage plus the prefix/denominator analysis from the cache post |
| What did the session cost? | Reconciled input, output, cache-write/read and other charged categories |
These metrics should be segmented by bounded dimensions such as model, endpoint, operation and outcome. Histograms are useful for latency distributions; one average hides long waits. Keep workload and sample windows visible when making comparisons.
Cost needs an unknown state
For each physical attempt, preserve the provider’s raw usage and pricing version. Normalize categories only with a documented mapping. Do not add overlapping categories twice, and do not assume missing usage means a free request.
A session total needs retries, compaction calls and any warm-up requests. Some charges may be reported later or require reconciliation. Display known cost and unresolved attempts separately until the evidence supports a complete total. Pricing counters are not proof of physical FLOPs or KV residency.
Which open-source stack?
For this personal lab, I would begin with OpenTelemetry instrumentation and a local backend selected against these requirements.
SigNoz brings traces, logs, metrics, dashboards and alerts into one observability platform. That makes it a reasonable candidate for the broad workflow I want to investigate. Grafana Tempo is another trace-storage route within a wider metrics/logs stack. Langfuse adds LLM-oriented tracing, prompt management and evaluation workflows. They solve overlapping but different problems; a product-name comparison is not functional parity with every Datadog feature.
This is a proposed local evaluation, not an installed or benchmarked stack. First verify trace propagation, required fields, ARM64 deployment if used, and the ability to reconcile one synthetic turn. Add dashboards after those records are trustworthy.
Where the current implementation stops
The public runtime currently saves lifecycle events and completed responses. Its provider adapter buffers the complete response; its SSE endpoint replays persisted events. That is useful evidence about the agent loop, but it is not model-token streaming or a first-token measurement.
The next implementation step is to capture actual content events and physical-attempt spans while preserving completed results and cancellation/error behavior. Then replay fixed workloads and inspect a known failure case before comparing backend overhead.
What gets measured should help us decide what to change. A collection of attractive charts that cannot explain one slow, expensive turn has not reached that point.