Inference economics

1The baseline, measured rather than assumed

First word came back in about 1.5 seconds. The full answer took about 24. The call was blocking and non-streaming with no length cap, so every one of those 22 intermediate seconds was spent generating text the user could already have been reading. It was not a frontend problem and it was not a model problem.

Before doing anything clever I tried the boring option: a cheaper tier of the same paid family. The mini tier took 22.7 seconds to first word and the nano tier took 59.8, both more expensive per answer than the model I was trying to replace, at $0.022–0.029. The reasoning tiers spend their budget thinking about a short structured task that does not need it. That was the end of the cheap-paid-model idea.

2Splitting the work up made it lie

Running four parallel calls, each responsible for one section of the answer, cut wall time to about eight seconds. It also made the model invent things. Asking a call specifically for 'credit insights' produces credit insights whether or not any apply: it invented ineligible credits in nine runs out of ten, and quality fell to 5.5 out of 10.

A single call sees the whole record and can decline a section. A call whose entire job is one section has no way to return nothing. I have stopped treating parallel decomposition as a free latency win.

3The gap was facts, not weights

Open models out of the box scored 5 to 6 out of 10 against a frontier baseline of 8.5 to 8.7. A 30B open model, a 70B one and a 32B reasoning model all landed in the same band, which was the first clue: when three different architectures fail identically, they are missing the same information, not the same capability.

So I put five verified reference facts into the prompt — computed values, not instructions. The open 30B model went from 5 to 8 and drew level with the frontier model. The same five facts were worth a full judge point on every model I tested, including the expensive one.

I also tried the other thing everyone tries, which is writing behavioural rules into the prompt. Telling the model how to behave was worth essentially nothing: the best rules variant scored 6.05 against 7.25 for facts alone. Stacking rules on top of facts made it worse than facts alone, at 6.1. I do not have a satisfying explanation for why adding correct instructions degrades a grounded prompt, only ten runs that say it does.

4Serving it: the cache is the product

Both serving engines I tested work. vLLM gave a warm first token at about 114 milliseconds, SGLang about 146. The engine choice barely matters; the prefix cache matters enormously. Warm versus cold is roughly an 18× difference on first token, and that is the entire reason self-hosting is cheap — the long stable part of the prompt is paid for once.

Self-hosting beats the API above about 270 answers an hour. Measured throughput ran near 2,880 an hour, roughly ten times past the break-even, so this is not a marginal call. Batched cost came to $0.00127 an answer.

The managed-compute option is worth knowing about: about 15% more per answer than running the machine myself, 1.6× faster wall time, and no VM to operate. If I were doing this inside a team rather than alone I would probably take that trade.

5The judge that corrected me

For most of the project the quality scores came from the model judging its own output. When I replaced that with two independent judges, every absolute score fell by 1.3 to 1.5 points. The self-judge had been inflating everything uniformly.

Two things survived: the ranking between options was unchanged, and the two independent judges agreed with each other within half a point. One of them also caught a fabricated number that the self-judge had passed. That single catch is why the final architecture computes numbers in code.

6Where it ended up

The final recipe reached 7.95 against the frontier model's 8.0 on the independent judge, with zero ungrounded numbers across the run, 6.1 seconds wall and 614 ms to first token. Against the original 13.9 seconds and 1.5 seconds, and at roughly an eighth of the cost per answer.

One more finding that only showed up because I measured the expensive model too: a terse prompt cut its wall time by 32% and its cost by 33% with quality held at 8.0. Output length was the only real lever on the paid API. Everything else I tried there was noise.

And one candidate disqualified on its own numbers. A newer model scored best in test at 8.0, and then turned out to emit a median of 1,926 hidden reasoning tokens, which pushed first token to 18 seconds and real cost 30% above the model it was meant to replace. Benchmarks do not bill you; invoices do.

The architecture I would actually ship is the least glamorous one I tried. A rule engine computes every number, category and priority in about 10 milliseconds, and the model only writes prose around values it was handed. It never sees the raw record. 8.5 out of 10 with zero ungrounded numbers in ten of ten scenarios, around $0.0062 a call, four to four-and-a-half seconds. A model that cannot see the numbers cannot make them up.

7What is still open

The hybrid mode — giving the model a short three-year summary instead of the full raw record — found an income trend in two runs of two, where the raw record found it in zero of two at five times the tokens. That is two runs. It is suggestive and it is not evidence, and the thin-record case still needs data.

8Evidence

Every step was written down at the time as a claim, its evidence and a verdict — AI Insights: Latency & Cost Optimization (17 parts). The discarded runs are in there too.

Where the work lives:

• a private folder — the latency and cost lab.
• ~/company/blog/content — the 16-post series this produced.