Performance

Raw latency doesn't matter. Latency under load does.

On its own, Vidai adds 0.58 ms median at up to 47,200 calls a second on one machine. Across a real network it held 21,803 calls a second at 1.95 ms median, network time included, while Python-based proxies peaked under 200.

Latency as load rises slow fast low traffic 21,803 calls/s Python-based proxy Vidai, flat at the peak
Everyone is fast at low traffic. What matters is who stays flat at the peak.

Three tests, three questions

One story, measured three ways.

Each test answers a different question, so each figure is quoted with its own set-up. Don't compare a figure from one test with a figure from another.

On one machine

Up to 47,200 calls a second, with 0.58 ms median added

How much does Vidai itself add to a call?

Vidai, a mock provider and the load generator on one machine, so no network time is counted: the figures are Vidai's own. 32 connections, four 15-second runs, every call recorded to PostgreSQL, guardrails and TLS off.

Across a network

21,803 calls a second at 1.95 ms median, network included

What does a real deployment see, network included?

Load sent across a real network to Vidai on one 2013 Intel Xeon E3-1240 v3 (8 cores), with VidaiMock as the provider, so every figure includes network time. The test network reached its limit before Vidai did.

Head to head

13 to 17 times LiteLLM's throughput, 3.5 to 3.9 times Portkey's

How does Vidai with governance on compare with gateways doing less?

Vidai with authentication, rate limits and policy on, against LiteLLM and Portkey in minimal pass-through mode, on the same harness at three load profiles.

47,200 is the highest rate tested, not a guaranteed limit. Python-based proxies on the same harness peaked under 200 calls a second, at 197 ms median. The scripts are open source, so you can run the tests yourself: benchmark scripts. Results on your infrastructure will differ.

Why load is now the only number

A benchmark used to be a vanity metric. Agents made it a survival metric.

In the chat era, your AI traffic was paced by people: one prompt, one call, human typing speed as the ceiling. Agentic AI removes that ceiling. One task now fans out into ten or more autonomous calls, and the traffic through your in-path layer is climbing 10× and still rising.

The layer in front of every AI request is no longer handling chat-pace load. It is handling agent-pace load, and its speed under that load is now your AI's speed.

See what 10× looks like at your scale →
1 Human era paced by people 2 Chat era paced by people 10× Agent era paced by the agent and still growing
AI calls per task. 10× is the conservative figure (IDC); measured ranges run higher.

What it means for the business

Fast enough that nobody notices it's there.

Every 100 ms of added latency measurably reduces engagement. Vidai sits in the path of every AI request and adds under a millisecond of its own: 0.58 ms median, measured on one machine with no network in between. It doesn't become the bottleneck when traffic spikes 10×.

Performance is a property of being a control plane, not a feature you tune. What's a control plane? →

Across a network

21,803 calls a second, network included.

Load sent across a real network, so every figure below includes network time. The same harness, three kinds of AI gateway, pushed to their peak. The median tells you about a quiet day; p95 and p99 tell you what your slowest users feel at the peak.
21,803calls a second on one node, Vidai still stable
1.95 msmedian latency at that rate, 40 ms at p99
197 msmedian for a Python-based proxy at its peak, under 200 calls a second
Peak throughput and latency for three kinds of AI gateway on the same harness
GatewayPeak RPSMedianp95p99Status
Vidai (Rust) 21,8031.95 ms23.5 ms40 ms Stable
Python-based proxies < 200197 ms1.3 s6.0 s Past its limit
Cloud-managed gateways ~ 640100 ms395 ms1.2 s At its limit

Method: a single legacy 8-core node, an Intel Xeon E3-1240 v3 from 2013, chosen so the figure is a floor, not a best case, with VidaiMock as the provider. 21,803 is where the test network reached its limit, not Vidai. Full methodology and reproduction steps are in the technical write-up.

Per-core scaling

Why a single node carries so much.

The whole-machine number above is the headline. The number underneath it is the more telling one. Each core in the test added roughly 7,000 requests per second of headroom in a linear scaling curve, up to about 29,000 RPS in total before shared-hardware contention (NIC, cache) prevented further increase. On the same harness, Python-based AI proxies bottlenecked around 75 to 110 RPS per core: a per-core efficiency gap of roughly 94×.

That gap is the reason a fleet of Python nodes does not bridge the difference: 94 of them, behind a load balancer, replicate the bottleneck 94 times. The fastest path through this is not more nodes, it's a faster runtime.

94×

per-core throughput gap

~7,000 Vidai ~75 to 110 Python-based proxy
Requests per second per core. Same benchmark harness.

Infrastructure footprint

What decides your node count: headroom, or a ceiling.

Nobody runs 21,803 requests per second sustained; that is a stress-test peak. The number that matters is what one node clears with room to spare. With that much headroom, real traffic, even agent-pace traffic, fits inside one node, so you add nodes for redundancy, not to keep up. The slower the runtime, the sooner each node redlines, and then node count is set by load, not by choice.

Adding nodes is never free

"Just load-balance it" is a real answer, with a real bill: each node is compute, load-balancer capacity, and one more thing to patch and page for. The question is whether you add nodes by choice or by necessity.

Vidai nodes scale for HA, not headroom

A single node already carries the load with margin. The second node is the one you fail over to, not the one that stops you redlining.

A redlining node has no margin for the spike

When traffic jumps 10× for an hour, headroom absorbs it. A node already near its ceiling has nowhere to put it, so the fleet has to be sized for the peak, all the time.

Efficiency per node is the lever

Cut what one node clears and you multiply the fleet, the load balancer and the on-call surface with it. Agent-pace traffic only sharpens that maths.

See how it scales out →

Why each node carries more

The headroom is a property of a Rust hot path.

The per-node margin that lets you scale by choice is not a tuning trick. It falls out of language and architecture choices that hold the same shape under 10× the load.

No garbage collection

No tail-latency spikes when the collector runs mid-request.

No GIL

Real multi-core throughput on every node, not a single-threaded process that scales only by adding more processes.

Zero-copy networking

The request isn't re-serialised three times on the way through.

~25 MB engine

Small footprint, low hardware bill, trivial to replicate.

High availability, today

Run several Vidai servers behind your load balancer, with autoscaling. Add a server for resilience, not because one ran out of room.

More on operations →

How we start together

Want the benchmark methodology.

A 20-minute technical walkthrough, including how to reproduce the numbers. Then, if it fits, a two-week pilot on your own traffic.
  1. 01

    Pick one app or team

    Point it at the Control room. One configuration change; nothing else changes.

  2. 02

    Watch for two weeks

    Every call priced, checked and recorded. Rules start in observe mode.

  3. 03

    See the evidence

    What it cost, what it caught and what it would have stopped, from your own traffic.