AI Insights: Latency & Cost Optimization · part 7 of 17
Throughput and the season cost model
What one answer actually costs at millions of records per season
The question
Fairly compared, what does an answer cost on the paid API vs a self-hosted GPU at production volume?
Self-hosting beats the API from ~270 answers/hour, and measured throughput runs about 10x past that break-even. The gap is large, not marginal.
Effective per-token rates
The GPU does input and output alike, so Qwen's figure is a single blended rate — roughly $0.33/M once the GPU is kept busy (about $2.33/M serving one answer at a time). Compare that to GPT-5.1's $10 output rate and the cost gap is explained. Output tokens dominate GPT-5.1's bill: the ~1,300 written tokens cost far more than the 2,600 read tokens — another reason to cap response length.
| Cost per 1M tokens | Input | Output |
|---|---|---|
| GPT-5.1 (paid API) | $1.25 | $10.00 |
| GPT-5.1 with prompt caching (repeated prompt) | $0.125 | $10.00 |
| Qwen 30B on vLLM (self-hosted, at scale) | ~$0.33 blended | ~$0.33 blended |
Season cost (millions of records, PAYG GPU pricing)
| AI calls per return | Total calls | GPT-5.1 paid API | Qwen self-hosted PAYG | Qwen spot |
|---|---|---|---|---|
| 1 | 12M | ~$164,000 | ~$15,000 | ~$4,000 |
| 5 | 60M | ~$819,000 | ~$76,000 | ~$21,000 |
| 10 | 120M | ~$1,640,000 | ~$153,000 | ~$42,000 |
| 15 | 180M | ~$2,460,000 | ~$229,000 | ~$63,000 |
The shape matters more than any single number: the API charges per call, so the bill grows linearly with usage forever. Self-hosting charges per GPU-hour no matter how many answers the hour produces — the more it's used, the cheaper each answer gets. One caveat: self-hosting carries ops cost the API doesn't (on-call, model updates, security review, capacity planning). It doesn't change the conclusion — the gap is ~10x, not 1.4x — but it belongs in the picture.