- Challenge
- The engineering problem we set ourselves: let many teams use language models freely, while spend stays predictable enough to plan around — and without putting an approval queue between a developer and their editor. The two usual answers both fail. Open access means the bill is a surprise at the end of the month. Central gatekeeping means nobody uses the models at all. We wanted a third answer, and we wanted it measured rather than argued.
- Solution
- We designed and built a single controlled entry point that every request passes through before it reaches the model provider. Each key carries a spending limit; each team carries a budget with a hard cut-off, so when the budget is gone all of that team's keys stop at once rather than after somebody notices. Usage is counted accurately even for streamed responses — the case where naive metering quietly under-counts. Cost, tokens, and latency are traced per call, so the question "what did AI cost last month, and for whom?" has an answer with a breakdown behind it. We then ran the whole thing against the tools engineers actually work in, because a gateway nobody can point their editor at is a gateway nobody uses.
- Result
- Budget overshoot verified at under 1% against real budgets. Per-team hard cut-off confirmed to fire as designed. The gateway's added latency measured at roughly two seconds, with clear guidance on which traffic is worth routing through it and which is better sent direct. Streaming token counts confirmed accurate, and the whole path validated against real day-to-day developer clients.
R&D
An LLM gateway that keeps multi-team AI spend under control
Give a whole engineering organisation access to language models and two things happen: the work gets faster, and nobody can say what it will cost. We set ourselves the problem of fixing the second part without slowing down the first — and built a gateway that holds a budget per team, cuts access off when it's spent, and traces every request end to end.
This is R&D we ran, not a client production system. It is the capability we deliver — the design, the controls, and the numbers below are ours and measured.
The work
What we set out to prove.
Supporting detail: an LLM proxy (LiteLLM-class) in front of AWS Bedrock, with PostgreSQL for budget state and Langfuse for cost, token, and latency tracing; containerised and deployable into a customer's own cloud account. The same pattern applies in front of other providers or self-hosted open-weight models — it is not Bedrock-only.
Wondering what this would cost to run — and to control — on your side?
Start a conversationOutcomes
What we measured.
Four numbers, all from our own R&D runs. We publish these because we measured them; we don't publish request volumes, savings figures, or team counts, because we didn't.
<1%
Budget overshoot
Verified against real budgets — the difference between AI costs you can plan around and AI costs you discover on the invoice.
Per-team
Hard spending limits
Budgets and limits per team and per key, with a hard cut-off: when a team's budget is gone, all of its keys stop at once.
~+2s
Added latency
The gateway's own overhead, measured — with guidance on which traffic to route through it and which to send straight to the model.
Full
Cost observability
Cost, tokens, and latency traced per call in Langfuse, including accurate counting on streamed responses.
Provenance: these results come from R&D we ran to production readiness, not from a system we operate for a client. Every figure above was measured in that work; nothing here is an estimate or a projection.
What this means for you
The same controls, on your infrastructure.
We built this before offering it, so the method was tested at our expense rather than yours. If your organisation is heading towards broad AI use, this is the shape of the work.
Spend that finance can plan
Budgets per team and per key, enforced rather than reported. The month's AI cost stops being a number you find out about after the fact.
Access without a gatekeeper
Developers keep working in their own editors and internal apps. The controls sit behind them, so adoption doesn't depend on anyone changing how they work.
One place to see and to switch
A single entry point means one audit trail, one set of access rules, and one place to change models or providers without touching every team's code.
Ready for a private deployment
The same gateway sits equally in front of a private cloud endpoint or self-hosted open-weight models, which is how a private LLM rollout stays governable as it grows.
Rolling AI out across more than one team?
Tell us how your teams use models today and what your finance and security people need to see. We'll come back with the controls your case actually needs — and say so if a gateway isn't one of them.