Explainer
What is AI infrastructure?
A model is a component, not a system. AI infrastructure is everything around it that makes the output reliable, affordable and safe to act on: the compute it runs on, the data it reads, the serving layer that exposes it, the orchestration that sequences its work, the memory that carries state, and the control layer that decides what it may actually do. This page explains each layer in plain English, says what vendors sell at each one, and sets out what agentic systems add on top.
The six layers
- Compute. GPUs, accelerators, or hosted inference bought by the token. The decision is rarely raw speed; it is cost per successful task, cold-start latency, and whether you can move providers without rewriting.
- Data. Object storage, pipelines, and an index built for retrieval — vectors for similarity, a graph or relational keys for relationships. Retrieval quality sets the ceiling on answer quality; no model recovers from feeding it the wrong documents.
- Serving. The API in front of the model: batching, caching, streaming, rate limits, fallbacks between providers, and cost accounting per request. This is the layer that turns unpredictable latency into a product-grade response time.
- Orchestration. The code that decides the sequence of calls, invokes tools, retries failures and stops runaway loops. In agentic systems this is where most of the behaviour lives, and where most of the bugs do.
- Memory and state. Durable context across steps and sessions: what happened, what is known, what has been superseded — with retention, tenancy isolation and deletion decided up front rather than retrofitted.
- Control and evidence. Isolation for untrusted execution, policy on which tools may be called with which arguments, and logs that reconstruct any past answer. This is the layer enterprise buyers audit, and the one most stacks are missing.
What changes when the system is agentic
Ordinary model serving is a request and a response. An agent plans, acts, observes and repeats — so the risk moves from the text it produces to the actions it takes. Three requirements appear that a plain inference stack never needed.
Isolation. An agent that writes and runs code needs an environment with no path back to production systems: ephemeral, network-restricted, and destroyed after use. Tool-call policy. Permission has to be enforced by infrastructure, not by an instruction in a prompt, because the prompt is exactly what an attacker influences. Provenance. Every step needs a record of the data read, the rule version applied and the decision taken, or the system cannot be reviewed after the fact.
Those three are set out in detail, with the architecture behind each, in the secure AI infrastructure hub.
What AI infrastructure companies sell
Buyers almost never purchase one platform covering all six. They assemble four or five vendors and write the glue, which is why the interfaces between layers — not the layers themselves — are where production failures concentrate.
Four questions that test a stack
- Can you reproduce any past answer, including the data and the rule versions that produced it?
- Can a forbidden action be blocked by policy, rather than merely discouraged in a prompt?
- Do you know the cost per successful task, not just the cost per token?
- Can you delete one customer’s data everywhere, including derived indexes and agent memory?
A stack that fails any one of these will pass a demo and fail an enterprise security review — which is usually where a promising AI product stalls.
Questions people ask about AI infrastructure
What is AI infrastructure?
AI infrastructure is everything a model needs around it to do useful work in production: compute to run on, storage and data pipelines to read from, a serving layer that turns a model into an API, orchestration that sequences calls and tools, memory and state so work carries across steps, and a control layer that enforces what the system is allowed to do. The model is one component. The infrastructure is the other nine-tenths, and it is where reliability, cost and risk actually live.
What infrastructure is needed for AI?
Six layers, in the order they usually bite. Compute — GPUs or hosted inference. Data — storage, pipelines and an index for retrieval. Serving — an API in front of the model with batching, caching and rate limits. Orchestration — the code that plans steps, calls tools and handles failure. Memory and state — durable context across sessions, with retention rules. Control and evidence — isolation, policy on tool calls, and logs of what happened. Most teams buy the first three and build the last three.
What infrastructure is required to support agentic AI?
Agentic systems add three requirements beyond ordinary model serving. Isolation, because an agent that writes and runs code must do so somewhere that cannot reach production. Tool-call policy, because the risk is not the text the model produces but the actions it takes. And durable memory with provenance, because a multi-step task needs to know what already happened and an auditor needs to know why each step was taken.
What are AI infrastructure companies?
They are the vendors selling those layers rather than the applications on top: hosted compute and inference providers, vector and graph database vendors, orchestration and agent-framework companies, evaluation and observability tools, and the runtime security layer that isolates execution and enforces policy on tool calls. The market is layered, and a buyer usually assembles four or five of them rather than buying one platform.
Is AI infrastructure the same as an AI platform?
No. Infrastructure is the set of capabilities; a platform is one vendor's bundle of several of them behind a single interface. Platforms trade flexibility for speed — quick to start, harder to move away from when one layer proves unfit. The practical test is whether you could replace the model, the vector store or the orchestration independently without rewriting the product.
How do you know if your AI infrastructure is adequate?
Four checks. Can you reproduce any past answer, including the data and rules that produced it? Can an agent's action be blocked by policy rather than by a prompt instruction? Do you know the cost per successful task, not just per token? And can you delete a user's data everywhere, including derived indexes and memory? A stack that fails any of those will pass a demo and fail an enterprise review.
How much of AI infrastructure should you build yourself?
Buy compute, serving and storage — there is no advantage in operating them. Build the layers that encode your product's judgement: how work is orchestrated, what is remembered, and what agents are permitted to do. Those three decide whether the product is defensible, and they are also the ones that are painful to change once customers depend on them.
Sources
- Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, NeurIPS 2015 — why the code around a model outweighs the model.
- Google Cloud, MLOps: continuous delivery and automation pipelines — the data, serving and pipeline layers in practice.
- Ray Serve documentation — batching, autoscaling and multi-model serving behaviour.
- NIST AI Risk Management Framework 1.0 — the govern, map, measure and manage functions an enterprise reviewer works from.
- OWASP Top 10 for LLM Applications — the failure classes the control layer exists to stop.
- NIST SP 800-207, Zero Trust Architecture — the isolation and policy model applied to agent tool calls.
Every claim above can be checked against these. The demand figures quoted elsewhere on this site are search-market data from Semrush, not traffic to the domains themselves.
The control and evidence layer — isolation, tool-call policy and audit trails — is set out asset by asset in the Secure AI Infrastructure hub, alongside what AI runtime security actually enforces.