AI Insights: Latency & Cost Optimization · part 11 of 17
insight27 July 2026cost: ~$4.20 (36-min bench, auto-teardown)
Foundry managed compute: Qwen without running the VM
Same model from the Azure Foundry catalog on managed H100, benchmarked end to end
The question
Can the winning Qwen recipe run on Azure Foundry managed compute — no VM, no vLLM ops — and what does that cost?
insight
Managed H100 is ~15% costlier per answer than the DIY A100 but 1.6x faster wall time — a workable option that removes VM ops.
Wall (single, V5)5.5 s1.6x faster than DIY A100
TTFT544 msgateway overhead; DIY = 92 ms
Cost/answer$0.00149@ $6.98/hr, conc 16
Throughput sweep660–4,680 /hr@ conc 1/4/8/16
Results (n=10, V5 prompt, H100 BF16)
| Metric | Foundry H100 (managed) | DIY A100 + vLLM | GPT-5.1 API |
|---|---|---|---|
| Wall time (single) | 5.5 s | 9.0 s | 15–24 s |
| TTFT | 544 ms (gateway) | 92 ms | ~1,500 ms |
| Quality (Opus/Sonnet) | 7.0 / 6.0 | 7.25 / 7.0 (FP8) | 8.0 / 7.5 |
| Ungrounded numbers (median) | 1.5 (BF16 regression) | 0 (FP8) | 0 |
| Cost per answer | $0.00149 | $0.00127 | $0.0136 |
| 180M-call season | ~$268K | ~$229K | ~$2.46M |
Deployment mechanics that cost a day
- The FP8 H100 template is dead — deployed 'Succeeded' twice but returned 500 forever. The BF16 1xH100 template works on first ping.
- Auth is the
api-keyheader —Bearer+ account key returns 401. - The
modelfield in requests must be the deployment name, not the model id. - Deployment templates are listed via
azure-ai-mlMLClient(registry_name=...).deployment_templates.list() withDeploymentTemplate._from_rest_objectmonkeypatched (SDK parse bug); deployment itself viaazure-mgmt-cognitiveservices==15.0.0b2managed_compute_deployments. - Direct HTTPS endpoint, no tunnel:
.../managed-deployments/<name>/v1/chat/completions(also/v1/messages). Managed-compute quota is separate from VM quota — 1xH100 fit the existing allowance.