Research
We upgraded vLLM from 0.16 to 0.17, kept the same flags, and performance got worse. Then we added one flag and latency dropped by 6x. Here is exactly what happened, why, and what you need to do if you run vLLM in production.
March 2026 | Dry Ground AI Research
Key Findings
Three configurations, same hardware, same model (Qwen3-30B-A3B-Instruct-2507-GPTQ-Int4), same 13-test quality suite. Here is what we found.
Every configuration scores 12/13 on our production benchmark suite. The same test fails in all three: multi-step-reason-1, a gross profit calculation. Classification, structured output, instruction following, code generation, email drafting, and summarization all pass cleanly. The version upgrade did not move quality in either direction.
Running vLLM 0.17.0 with --enforce-eager (the same flags used in v0.16) produces a small regression. Medium prompt latency went from 4,602ms to 5,151ms. Long prompt latency went from 29,130ms to 30,605ms. If you upgraded versions and kept your existing launch flags, you got worse performance without knowing it.
Switching to --performance-mode throughput in v0.17 enables CUDA graphs and torch.compile. Medium prompt latency drops from 4,602ms (v0.16) to 886ms (v0.17 throughput). Long prompt latency drops from 29,130ms to 4,817ms. The version upgrade alone does nothing. The flag does everything.
At 8 concurrent streams, v0.17 with --performance-mode throughput produces 714.3 aggregate tok/s compared to 167.3 for v0.16. That is a 4.3x improvement. The gap widens with concurrency. At 1 stream the improvement is 4.3x. At 8 streams it holds at 4.3x, meaning the scaling efficiency is consistent.
vLLM 0.17 throws CUBLAS_STATUS_INVALID_VALUE on CUDA 12.9 hosts. The fix is a single command before launching: unset LD_LIBRARY_PATH. This is in the v0.17 release notes but easy to miss. Without it, the server fails to start on affected hosts.
Numbers
All three configurations ran on the same dedicated 80GB GPU, same model weights, same 13-test quality suite.
12/13
Quality (all configs)
92.3% pass rate, same test fails in all three
886ms
v0.17 throughput latency
medium prompt (was 4,602ms in v0.16)
145.4
Long-gen tok/s
v0.17 throughput (was 24.0 in v0.16)
714 tok/s
Peak concurrent
v0.17 throughput, 8 streams
6.1x
Latency improvement
medium prompts, v0.17 throughput vs v0.16
4.3x
Concurrency gain
aggregate throughput at 8 streams
The Migration Pitfall
The performance gains in vLLM 0.17 come from two optimizations: CUDA graphs and torch.compile. These reduce per-request overhead significantly, especially for medium and long generation sequences where the kernel launch cost compounds across hundreds of decoding steps.
The catch: both optimizations are disabled when you pass --enforce-eager. That flag tells vLLM to skip graph compilation and run every operation eagerly. It was the correct choice in v0.16 because CUDA graph support was less mature. In v0.17, it is leaving most of the performance on the table.
If you ran v0.16 with --enforce-eager and upgraded to v0.17 without changing flags, you are running the new version at a performance disadvantage. PyTorch upgraded from 2.6 to 2.10 in this release, but the compile path is what delivers the gains. Eager mode bypasses it entirely.
Drop --enforce-eager and add --performance-mode throughput. The first startup takes longer as CUDA graphs compile and torch.compile warms up. Subsequent requests are significantly faster. On our hardware, medium prompt latency dropped from 4,602ms to 886ms.
Note that --performance-mode throughput optimizes for aggregate throughput and may slightly increase latency on the very first request after a cold start. For any sustained workload, it is the right choice.
Performance
Five iterations per prompt size. Temperature 0.0. Same model weights across all three runs. Latencies include network overhead from the HTTPS proxy.
| Configuration | Short lat. | Short tok/s | Med lat. | Med tok/s | Long lat. | Long tok/s |
|---|---|---|---|---|---|---|
| v0.16.0 (enforce-eager) | 471ms | 4.6 | 4,602ms | 22.2 | 29,130ms | 24.0 |
| v0.17.0 default (enforce-eager) | 454ms | 4.8 | 5,151ms | 20.7 | 30,605ms | 22.9 |
| v0.17.0 --performance-mode throughput | 280ms | 8.0 | 886ms | 118.6 | 4,817ms | 145.4 |
Short prompt note: At 16 max tokens, all three configurations are in roughly the same range (280-471ms). The CUDA graph benefit is proportionally smaller when the generation is short. The gains compound on medium and long generation where the compile path amortizes across many more decoding steps.
Concurrency
Simultaneous stream counts from 1 to 8. Values are aggregate tok/s across all concurrent requests. The throughput mode advantage holds and scales linearly.
| Concurrent Streams | v0.16.0 | v0.17 default | v0.17 throughput |
|---|---|---|---|
| 1 | 22.4 | 21 | 97.2 |
| 2 | 44.7 | 35.4 | 193.3 |
| 4 | 86.9 | 69.2 | 374.9 |
| 8 | 167.3 | 137.9 | 714.3 |
4.3x
throughput improvement at 8 streams
714 tok/s
peak aggregate (v0.17 throughput, 8 streams)
Near-linear
scaling from 1 to 8 streams across all configs
Quality
CUDA graphs and torch.compile are pure inference engine optimizations. They change how operations are scheduled and executed but do not affect model weights or the forward pass output. The scores confirm this: all three configurations pass exactly the same tests.
| Test | Description | v0.16 | v0.17 default | v0.17 throughput |
|---|---|---|---|---|
| classify-1 | Email classification | ✓ | ✓ | ✓ |
| classify-2 | Email classification | ✓ | ✓ | ✓ |
| classify-3 | Email classification | ✓ | ✓ | ✓ |
| structured-output-1 | JSON extraction | ✓ | ✓ | ✓ |
| structured-output-2 | JSON extraction | ✓ | ✓ | ✓ |
| structured-output-3 | Structured generation | ✓ | ✓ | ✓ |
| instruct-1 | Instruction following | ✓ | ✓ | ✓ |
| instruct-2 | Instruction following | ✓ | ✓ | ✓ |
| code-1 | Code generation | ✓ | ✓ | ✓ |
| email-draft-1 | Email drafting | ✓ | ✓ | ✓ |
| summarize-1 | Summarization | ✓ | ✓ | ✓ |
| reason-1 | Multi-step reasoning | ✓ | ✓ | ✓ |
| multi-step-reason-1 | Gross profit calculation | ✗ | ✗ | ✗ |
| Total | 12/13 | 12/13 | 12/13 |
The failing test: multi-step-reason-1 tests gross profit calculation with multiple inputs. This test fails on the model, not the inference engine. All three configs fail it identically, which confirms it is a model capability boundary, not an engine regression. This test passes on some model variants and fails on others.
Release Details
699 commits from 272 contributors. Four changes are material to production inference workloads.
The runtime upgrade provides the foundation for improved torch.compile performance. Most of the single-request latency improvement traces back to better compile output in 2.10, particularly for MoE attention patterns.
This is where all the gains live. CUDA graphs eliminate per-kernel launch overhead across decoding steps. torch.compile fuses operations that would otherwise execute as separate kernels. The combined effect is most visible on medium and long generation sequences where the overhead compounds.
vLLM 0.17 adds FlashAttention 4 as an optional backend. Our benchmark used the FLASH_ATTN (v3) backend, which was already in production. FA4 is worth evaluating separately, particularly for long-context workloads where its memory access patterns are most beneficial.
With --performance-mode throughput, vLLM compiles CUDA graphs at startup. On our hardware this adds roughly 2-3 minutes to cold start time. For a server that runs continuously, this is a one-time cost. For auto-scaling scenarios where pods spin up frequently, factor this into your readiness probe timing.
Known Issue
If your host runs CUDA 12.9 and you upgrade to vLLM 0.17, the server may fail immediately with a CUBLAS_STATUS_INVALID_VALUE error. This is documented in the v0.17 release notes as a known compatibility issue with CUDA 12.9.
The fix is to unset LD_LIBRARY_PATH before launching vLLM. Add this to your launch script or container entrypoint:
This removes any conflicting CUDA library path that causes cuBLAS to pick up the wrong version. Without this fix, the server fails before loading the model on affected hosts.
We hit this on first launch. The error message is not obvious about the cause. If you see CUBLAS_STATUS_INVALID_VALUE at startup and you are on CUDA 12.9, this is the fix before debugging anything else.
Production
The 6x latency improvement on medium generation is too significant to leave on the table. We are upgrading from v0.16 to v0.17 and dropping --enforce-eager in favor of --performance-mode throughput. Quality is identical, the CUDA 12.9 fix is a one-liner, and the performance gains are confirmed on our exact model and hardware.
Our benchmark used the FLASH_ATTN v3 backend. FA4 is available in v0.17 and the memory access improvements may benefit our longer-context tasks. We will run a focused comparison and publish the numbers.
Add "unset LD_LIBRARY_PATH" to launch script if running CUDA 12.9
Remove --enforce-eager from vLLM launch flags
Add --performance-mode throughput to launch flags
Expect 2-3 additional minutes on first startup (CUDA graph compilation)
Run quality suite to confirm 12/13 pass rate on your model
Verify latency improvement on your representative prompt distribution
Methodology
13 tests across 7 categories: classification (3 tests), structured output (3 tests), instruction following (2 tests), code generation (1 test), email drafting (1 test), summarization (1 test), and multi-step reasoning (2 tests). Each test has explicit pass/fail criteria. Same suite used across all model and version benchmarks we publish.
All three configurations ran sequentially on the same hardware with the server fully restarted between configs. No concurrent workloads during measurement. The only variable between Config 1 and Config 2 is the vLLM version. The only variable between Config 2 and Config 3 is the flag set.
vLLM 0.17 ships with significant improvements, but none of them activate with the default migration path (same flags as v0.16). If you benchmarked v0.17 with --enforce-eager and concluded the upgrade was not worth it, your conclusion was correct for that configuration and wrong about the version.
The gains live in --performance-mode throughput. The new PyTorch runtime and CUDA graph improvements are the mechanism. The medium-prompt latency improvement from 4,602ms to 886ms is the result. That number changes what is possible in latency-sensitive workflows.
This article tests vLLM versions, not models. For the broader picture on how we selected our production model, what the full benchmark suite covers, and how we compare across quantization configurations, see the inference benchmarks article.
Inference Benchmarks: Three Configurations, One ArchitectureBenchmarks conducted March 2026 on dedicated GPU infrastructure (80GB VRAM). Model: Qwen3-30B-A3B-Instruct-2507-GPTQ-Int4. Three configurations tested: vLLM 0.16.0 with --enforce-eager, vLLM 0.17.0 with --enforce-eager, vLLM 0.17.0 with --performance-mode throughput. Quality suite: 13 tests across 7 categories under production conditions. Network latencies include HTTPS proxy overhead (~50-100ms). Concurrency tested at 1, 2, 4, and 8 simultaneous streams. FLASH_ATTN (v3) backend used for all configurations.
We use cookies to improve your experience. Cookie Policy · Privacy Policy