Latency-critical AI
Your p99 is the number that loses you the auction.
In real-time bidding, trading and fraud scoring, every AI call sits inside a hard deadline measured in single-digit milliseconds. Vidai's overhead stays flat at the peak, while Python-based alternatives spike into seconds exactly when traffic is highest.
Who this is for
Four models, one auction, one clock.
Bid shading and pricing
Predicting the optimal clearing price in a first-price auction so the buyer does not overpay. The tightest budget on the path: it runs on essentially every impression.
CTR and CVR prediction
Scoring click-through and conversion likelihood at bid time. The model that decides whether the impression is worth bidding on at all.
Pre-bid fraud and IVT
Catching bots, data-centre traffic and click farms before money is spent. Gradient-boosted trees over session signatures, run as a parallel ensemble.
Traffic shaping (sell side)
On the SSP, filtering bid requests a buyer is unlikely to want before they ever go over the wire, saving bandwidth and compute at both ends.
The budgets above are per model, inside the wider auction window. They are the reason the layer in front of these models cannot add a millisecond it cannot account for. Trading-adjacent inference, fraud scoring and live product surfaces face the same shape of deadline.
Agents multiply it
One agent task, ten calls or more, all on the clock.
One task fans out
Time the governing layer adds to a whole ten-call agent task, network included.†
The overhead adds up
Ten calls in sequence pay the layer's cost ten times. A gap that looks small on one call becomes the difference between a task that fits its window and one that doesn't.
The tail compounds
If one call in a hundred is slow, a ten-call task meets at least one slow call about one time in ten. For agents, the p99 of a call becomes everyday behaviour for the task.
Parallel steps wait for the slowest
When an agent fans out, the step finishes only when its last call returns. Flat overhead at the peak is what keeps every branch predictable.
†Ten calls in sequence at the medians of our published network test: 1.95 ms for Vidai at 21,803 calls a second, 197 ms for a Python-based proxy at its peak, on one 2013 Intel Xeon E3-1240 v3 (8 cores). Method and full results
The real problem
It's not the median. It's the peak.
Measured on a single 8-core node, an Intel Xeon E3-1240 v3 from 2013, chosen so the figure is a floor, not a best case. At their own peak, Python-based alternatives ran at a 197 ms median.
The architecture trend
The hot path is moving off the network hop.
In the path, not another hop
Vidai is not a microservice your bidder calls over the wire. It sits in the request path as one compiled Rust engine, so governing the traffic does not cost you a round trip.
It is the path, not the model
Your bid-shading, CTR and fraud models stay yours, wherever they run. Vidai is the governed layer around them: routing, guardrails, cost and fallback in one pass, not seven.
No collector on the hot path
A Rust engine has no garbage collector, so there is no pause to land mid-auction. The variance a collector introduces is simply not in the system.
The honest line
Vidai does not run inside your bidding loop. It removes the middleware hops and the latency variance around your models, so the budget is spent on inference, not plumbing.
Hot path, cold path
Not everything belongs on the clock.
| Cold path, pre-computed | Heavy language work like contextual and brand-safety classification is too slow for the auction. It runs asynchronously and lands in an in-memory store. |
|---|---|
| Hot path, a fast lookup | At bid time the auction does a sub-millisecond read against that cache. The intelligence was computed earlier; the clock only sees a lookup. |
| Vidai governs the hot path | For the calls that must happen inside the window, Vidai is the layer in the path that keeps the overhead flat and deterministic, at the peak as at idle. |
| Fail fast, never hang | If a downstream model is slow, a circuit breaker trips and the path falls back rather than blowing the deadline. A late bid is a lost bid; a fast fallback is not. |
Why it holds the line
Deterministic by construction, not by tuning.
Rust hot path, no garbage collector
No collector pause mid-request. The overhead is the same at the peak as it is at idle, and that is the whole point.
One pass, not seven hops
Translation, routing, guardrails, cost and fallback happen in a single engine in the path, so there is no chain of libraries each adding its own millisecond.
In your path, on your infrastructure
No extra network hop out to someone else's cloud and back. The latency you measure is the latency you ship.
Cost control that doesn't cost latency
Per-call attribution and spend limits run inline at the same sub-2 ms, so governing the spend never slows the auction.
For platform engineering → Operations and fallback → Size your AI traffic →
How we start together
Put it on your critical path.
- 01
Pick one app or team
Point it at the Control room. One configuration change; nothing else changes.
- 02
Watch for two weeks
Every call priced, checked and recorded. Rules start in observe mode.
- 03
See the evidence
What it cost, what it caught and what it would have stopped, from your own traffic.