Architecture

Beyond the gateway: the governance-first control plane.

Every enterprise AI stack starts with a gateway. Eighteen months in, most are quietly rebuilding it as something stricter: a layer that doesn't just route each AI request but governs it, costs it and keeps a record of what it did. Network architecture has a name for that layer, and AI architecture is starting to use it: the control plane.

AI control plane

Cost ledger Policy enforcement Evidence record

AI gateway

RequestRouteModel

The gateway becomes one part of the control plane, not the whole of it.

When a gateway is enough

A gateway is the right answer to a smaller problem.

With one or two applications and one or two providers, and the question is how to route around an outage and unify the SDK, a gateway is exactly the right tool: LiteLLM, a thin in-house proxy or a managed routing service.

This isn't an argument that gateways are wrong. It's an argument that they're a different category of object from what comes next.

Past the threshold, a gateway still does real work, but it's one capability among several, not the whole story.

The threshold is quiet. It tends to be crossed when:

  • Two or more teams use AI through the same infrastructure, with different budgets.
  • A compliance, risk or audit function starts asking what the AI traffic actually did.
  • LLM spend moves from a rounding error to a line item with an owner.
  • A second model provider arrives, and choosing a provider becomes a policy question, not just a technical one.

The pivot

What an AI gateway doesn't carry.

A gateway answers one question: how does the request get to the right model? That question is solvable, and a gateway solves it. The questions a control plane answers sit a category above.
ConcernAI gateway or proxyAI control plane
Primary job Route requests Govern requests
Cost Logged, then forgotten Ledgered, enforceable, per request
Compliance posture Per-app paperwork, by hand Horizontal evidence, by construction
Performance ceiling Bound by middleware overhead Bound by physics
Operational outcome Tactical fix Infrastructure asset

The last row does the real work. A gateway is something a team owns and maintains: a tactical fix to a known routing problem. A control plane is something teams rely on and report into: an infrastructure asset with the same standing as a database or a service mesh, load-bearing, governed, and not part of any one team's surface area.

The argument

Three things a control plane does that a gateway doesn't.

01

The ledger is not a log

Most gateways log requests; some put a dashboard on the logs and call it cost visibility. That's observability, not governance. A control plane treats cost the way a financial system treats transactions: every request is charged to a budget, every budget has a holder, every holder has a threshold, and the threshold is enforced inline, before the next request goes through.

"We logged the spend" is a report Finance has to chase. "The spend stopped when the budget was hit" is a layer they can rely on.

Cost governance →
02

Compliance is horizontal, not per app

When each team's app carries its own guardrails, audit log and access policy, governance is per-app paperwork. Every team does it slightly differently, auditors get a different artefact for every system, and a policy change means touching every codebase that ever made an AI call.

A control plane gives one boundary, one set of rules and one evidence stream. When ISO/IEC 42001 becomes a procurement requirement, the question is "show me the control plane's evidence", not "did every app implement the controls".

ISO/IEC 42001 →
03

Performance is a property, not a tuning exercise

Anything in the path of every AI request can become the new bottleneck. A gateway written in Python carries overhead the runtime can't remove: garbage-collection pauses on the hot path, the GIL, repeated serialisation between middleware layers, network hops in and out of someone else's cloud.

A control plane built as a hot-path Rust process holds a deterministic overhead under 2 ms at the peak, where Python alternatives spike into seconds just as traffic is highest. Performance follows from the architecture you chose.

The benchmark →
21,803requests a second on a single node, our published stress-test peak
1.95 msmedian at that peak
Rusta single hot-path process, not layers of middleware

The benchmark is there as evidence of the architecture, not as a boast. One 8-core node, a stress-test peak rather than a sustained rate. Method and full results

The pivot, once more

Same SDKs. A different category of object underneath.

Moving from gateway to control plane isn't a rewrite. Applications keep using the OpenAI SDK, the Anthropic SDK, the Google ADK or whatever they were written against. What changes is what the SDK talks to: not a thin proxy one team maintains, but a layer every team relies on. Translation happens inside the control plane; the application is unaware.

So the most common objection, "we already have a gateway and can't afford to rewrite every app", is misframed. The applications don't change; the boundary they speak to does. The gateway was a tactical implementation of one job a control plane does. The control plane absorbs it and adds the ledger, the policy and the record the gateway never had.

Teams eighteen months into AI in production aren't, in general, arguing about whether a control plane is the right idea. They're deciding whether to keep building one badly themselves, or to buy one.

Vidai Control room · Keys
One key in Vidai: who it acts for, its translation, rate limit, the models it may call and its routing.
One key in Vidai: who it acts for, its translation, rate limit, the models it may call and its routing.

How we start together

Start with one team, on your own AI traffic.

After a 20-minute demo, we agree a two-week pilot with you: one app or team, your traffic, your infrastructure.
  1. 01

    Pick one app or team

    Point it at the Control room. One configuration change; nothing else changes.

  2. 02

    Watch for two weeks

    Every call priced, checked and recorded. Rules start in observe mode.

  3. 03

    See the evidence

    What it cost, what it caught and what it would have stopped, from your own traffic.