What runs underneath an inference API?

Two providers can accept the same JSON request and run very different systems underneath it.

An OpenAI-compatible endpoint establishes an interface. It does not tell us which scheduler, kernels, cache implementation or hardware served the request. Treating the interface as an engine label hides the part of the system we are trying to understand.

Begin with responsibilities

A request enters through an API or gateway, is admitted and scheduled, runs model operations, reads or allocates state, and returns a result or stream. Some systems combine these responsibilities; others put routing and orchestration around the model workers.

Before asking whether a deployment uses vLLM or SGLang, ask which responsibility the answer is supposed to explain.

The layered view is conceptual. Its boxes are not a claim that every deployment runs a separate service for each one.

Allocation and reuse solve different problems

An allocator decides where KV blocks fit in memory. A prefix matcher decides which previously computed state can be reused. An eviction policy decides what to discard under pressure. A scheduler decides which requests or portions of requests run next.

These decisions interact, but they are not synonyms. Paged allocation can reduce fragmentation without guaranteeing that the next request reaches matching cached state. A good prefix match can still require restoration from a slower tier.

vLLM’s Automatic Prefix Caching guide describes reusing KV for shared prefixes. SGLang’s RadixAttention account illustrates reuse, branching and eviction in a radix-tree design. These are mechanisms to inspect, not a promise that two configurations have identical performance.

What separates serving workers?

Prefill/decode separation adds a compatible-state transfer problem. The receiving worker needs the state in a layout and configuration it can execute. A transfer connector supplies a path; it does not by itself solve all routing, retention and failure policy.

vLLM documents disaggregated prefilling and connectors, including integrations with external KV systems. SGLang documents PD disaggregation and routing/transfer options. Support is specific to engine version, model architecture, parallelism and backend.

A feature being present in documentation is a starting point for a deployment test. It does not establish that an arbitrary combination is supported or that moving state costs less than recomputing it.

Where is retained state shared?

SGLang HiCache distinguishes GPU, host and configured storage tiers. Its private per-instance tiers should not be mistaken for a shared store across the whole deployment. Reuse beyond a worker needs the right backing store, namespace and compatibility.

The DeepSeek article makes the same distinction from another direction: shared storage, transfer and model execution each have their own job. A full serving stack combines those jobs with load balancing and operational policy.

Which companies use which engine?

Public sources can answer narrow versions of that question. Baseten’s engine chapter describes multiple engine choices, including TensorRT-LLM, vLLM and SGLang. DeepSeek’s DeepInfer release describes its own engine and its ancestry. Neither is a reason to assign one engine to every endpoint, model and date from a provider.

For an undisclosed OpenAI, Anthropic or Gemini request, I do not have sufficient evidence to name the current internal engine. Foundational research, compatible APIs or familiar response fields do not establish that inventory.

My working hypothesis is that large providers customize or mix components by workload. Public examples make that plausible; applying it to an undisclosed system is still an inference. A useful comparison table should contain unknowns rather than fill every cell with a confident guess.

What should a small experiment choose?

For the personal lab, the narrowest useful path is one model, one engine and one explicitly tested cache behavior. Record versions, hardware, numerical format, input/output lengths, concurrency and the latency objective. Include a relevant failure such as an incompatible state transfer or an evicted prefix.

That produces evidence we can compare. A broad plugin framework or a list of provider logos would not answer which mechanism helped the workload.