---
title: The 11 Layers of the AI Inference Stack: From Prompt to Silicon
slug: 11-layers-ai-inference-stack
canonical_url: https://blog.juscode.co/11-layers-ai-inference-stack
published_at: 2026-07-02T09:00:00+00:00
author: jusCode
tags: inference, infrastructure, ai factory
tldr: Inference stack is ambiguous. Follow one request from prompt to silicon and it passes through 11 distinct layers. Observability is a vertical across all of them, not a layer. You do not need to own every layer; you need to know where your line is. Ours sits just above the inference engine.
key_takeaways:
  - Inference stack is ambiguous: infra teams mean everything below the serving engine, app teams mean everything above it.
  - The fix: follow one request from prompt to silicon, and it passes through 11 distinct layers.
  - Observability is not a layer; it is a vertical that runs down the side of all 11.
  - You do not need to own every layer. You need to know where your line is; ours sits just above the inference engine.
---

Say "inference stack" to three engineers and you get three different buildings. The GPU engineer hears kernels and batching. The application engineer hears gateways and routers. The person paying hears cost per token. The fix: follow one request from prompt to silicon. It passes through 11 distinct layers.

## The 11 layers, top to bottom

Above the inference line (the harness, layers 01 to 07):

1. Application / Client. The product surface a human hits enter on (jusCode, jusInfer, Claude, ChatGPT). The only layer users see.
2. AI Gateway. One front door: auth, quotas, spend, safety, audit. Where bad requests die cheaply.
3. AI Router. Picks the model per request by cost, latency, and capability, with fallbacks.
4. Orchestrator. Multi-step logic: agent loops, tool calls, retries, chaining, and their failure modes.
5. Context Layer. What the model actually sees: system prompt, RAG, memory, trimmed to fit.
6. Cache Engine. Never compute the same thing twice: semantic, prefix, and KV caching. Often the highest-ROI layer.
7. Serving Engine. The last layer that thinks in requests: queueing, scheduling, batching, autoscaling.

Below the inference line (commodity power, layers 08 to 11):

8. Inference Engine. Runs the forward pass: vLLM, TensorRT-LLM, SGLang. Increasingly a solved problem you download.
9. AI Models. The weights: base models, quantized variants, LoRA adapters. Interchangeable from the router's view.
10. Compute Runtime. Kernels, compilers, drivers: CUDA, ROCm, Triton.
11. Hardware. GPUs, TPUs, HBM, interconnect. Everything above exists to keep this busy.

Observability, evals, and tracing are not a layer. They are a vertical that spans all 11.

## Where we build

At Kalmantic our line sits just above layer 08. Below it is commodity power, available to everyone on identical terms. The seven layers above are the harness, where cost, speed, and reliability are decided. We rent the power and build the harness.

## The AI factory floor plan

Every factory lives or dies on four numbers, and each layer moves at least one: throughput (tokens per second), unit cost (cost per token), lead time (time to first token), and utilization (GPU busy percent). Two lessons: most teams over-invest below the line and under-invest above it, and your line is a strategy, not a default. Instrument all eleven regardless, because observability is the one thing you can never outsource.

## References

1. Yu et al., 2022. Orca: iteration-level scheduling, the continuous batching that defines layer 08. OSDI 22.
2. Kwon et al., 2023. PagedAttention and vLLM. SOSP 23. arXiv:2309.06180.
3. Leviathan et al., 2023. Fast inference via speculative decoding. ICML 23. arXiv:2211.17192.


## FAQ

### Do I need all 11 layers on day one?

No. A prototype is layer 01 calling a managed API, two layers, and it is fine. The stack fills in as scale forces it: the gateway with the second team, the router with the second model, the cache with the first scary invoice. The mistake is not missing layers; it is not knowing which one your current pain lives in.

### Isn't observability a layer?

It is a vertical. Tracing, evals, and cost attribution have to span all 11 layers to be useful. A trace that stops at the gateway cannot tell you whether latency came from a cold cache, a queued scheduler, or a slow engine. Draw it down the side, not in the stack.

### Where do agents fit?

Agents are an orchestrator (04) pattern that leans hard on the context layer (05) and multiplies load on everything below. One agent task can mean fifty model calls, which is exactly why routing, caching, and scheduling stop being optional once agents ship.

### Why draw the line above the inference engine?

Because that is where differentiation ends today. Engines, weights, and silicon improve on their own schedule and are equally available to you and your competitors. The seven layers above are where your product's cost, latency, and reliability are decided. That is where we choose to build.