We tested OpenAI's open-weight flagship against our 30B production model on real business tasks. The 30B won on quality, matched on speed, and costs less than half to run.
February 2026 | Dry Ground AI Research
Key Findings
We ran OpenAI's GPT-oss 120B through the same benchmark suite we use for every model evaluation. 120 billion parameters, 2x A100 GPUs, head-to-head against our 30B production model on a single GPU. Here is what we found.
GPT-oss misclassified a contract review email as "urgent" instead of "action_required." For production email triage, that distinction matters. One wrong label multiplied across thousands of emails changes how a team prioritizes their day.
GPT-oss runs at 27.1 tok/s on 2x A100 SXM GPUs. Qwen3 hits 28.0 tok/s on a single A100 PCIe. Similar speed per token, but GPT-oss needs twice the GPUs. At $2.78/hr vs $1.19/hr, the economics do not work for production deployment.
Both are mixture-of-experts models. GPT-oss routes to 4 of 128 experts per token. Qwen3 activates 3B of 30B total parameters. Qwen3 achieves the same output quality with a fraction of the compute footprint.
GPT-oss ships with mxfp4 weights designed for H100 and H200 GPUs. On our A100s (Ampere architecture), vLLM uses the Marlin backend as a fallback. Native hardware support would improve throughput, but the quality gap remains regardless of GPU generation.
GPT-oss scores well on MMLU and HumanEval. It falls short on structured output precision. If your workload is chat and general reasoning, GPT-oss is competitive. If it is classification and extraction at scale, smaller specialized models win.
Benchmarks
Tested February 26, 2026. GPT-oss 120B running mxfp4 on 2x A100 SXM via Marlin backend. Qwen3 running GPTQ-Int4 on a single A100 PCIe.
10/11
Quality Score
GPT-oss misses 1 classification
27.1
GPT-oss Throughput
tok/s medium (2x A100 SXM)
28.0
Qwen3 Throughput
tok/s medium (1x A100 PCIe)
$2.78/hr
GPU Cost (GPT-oss)
2x A100 SXM 80GB required
$1.19/hr
GPU Cost (Qwen3)
1x A100 PCIe 80GB
120B vs 30B
Parameters
4/128 active vs 3B active
Performance
All latencies include approximately 50-100ms network overhead from RunPod HTTPS proxy. Five iterations per prompt size, temperature set to 0.0.
| Prompt Size | Qwen3-30B (GPTQ-Int4) | GPT-oss 120B (mxfp4) | Delta |
|---|---|---|---|
| Medium (128 tok) | 3,785ms / 28.0 tok/s | 4,731ms / 27.1 tok/s | -3% tok/s |
| Long (700 tok) | 23,112ms / 30.3 tok/s | 24,917ms / 28.1 tok/s | -7% tok/s |
| Concurrent Streams | Qwen3-30B (1x A100) | GPT-oss 120B (2x A100) | Delta |
|---|---|---|---|
| 1 | 27.2 | 26.3 | -3% |
| 2 | 56.2 | 37.9 | -33% |
| 4 | 109.3 | 102.4 | -6% |
| 8 | 205.8 | 202.1 | -2% |
The concurrency story: GPT-oss 120B on 2x A100 roughly matches Qwen3 on 1x A100 at every concurrency level. The exception is 2 concurrent streams where GPT-oss drops to 37.9 tok/s vs 56.2 tok/s. Two GPUs doing the work of one, at more than double the cost. That is the efficiency equation in a single number.
Quality
GPT-oss passes 10 of 11 tests. The failure is telling: a classification task where the correct answer is "action_required" but GPT-oss returns "urgent." For email triage at scale, that type of error compounds.
| Test | Description | Qwen3 | GPT-oss | Qwen3 Lat. | GPT-oss Lat. |
|---|---|---|---|---|---|
| json-extract-1 | Contact info to JSON | ✓ | ✓ | 2,032ms | 4,097ms |
| json-extract-2 | Meeting notes to JSON | ✓ | ✓ | 3,512ms | 12,524ms |
| classify-email-1 | Urgency classification | ✓ | ✓ | 475ms | 2,464ms |
| classify-email-2 | Newsletter classification | ✓ | ✓ | 309ms | 1,700ms |
| classify-email-3 | Action required | ✓ | ✗ | 675ms | 5,224ms |
| summarize-1 | Financial summary | ✓ | ✓ | 2,388ms | 4,094ms |
| instruct-1 | Format following (5 bullets) | ✓ | ✓ | 3,597ms | 6,145ms |
| instruct-2 | Arithmetic (strict format) | ✓ | ✓ | 619ms | 3,360ms |
| reason-1 | Capacity calculation | ✓ | ✓ | 411ms | 3,910ms |
| code-1 | Python function gen | ✓ | ✓ | 8,686ms | 18,334ms |
| email-draft-1 | Concise email draft | ✓ | ✓ | 1,459ms | 6,784ms |
The latency gap: GPT-oss is 2x to 9x slower per task compared to Qwen3. The biggest gap is reasoning (3,910ms vs 411ms) and classification (5,224ms vs 675ms). Even on tasks GPT-oss passes, it takes significantly longer to get there.
Architecture
Both GPT-oss and Qwen3 use mixture-of-experts routing to keep per-token compute manageable. The difference is in scale and efficiency. GPT-oss activates roughly 3x the parameters per token for the same quality result on our benchmark suite.
GPT-oss routes each token to 4 of 128 total experts. That means roughly 8-10B parameters activated per forward pass. Qwen3 activates 3B. For chat and reasoning, the extra capacity may help. For structured output, it does not.
At $2.78/hr for 2x A100 vs $1.19/hr for 1x A100, GPT-oss costs 2.3x more per hour with similar throughput. Per-token cost (including GPU amortization) makes GPT-oss roughly 2.3x more expensive for the same work.
GPT-oss ships with mxfp4 (microscaling FP4), a quantization format that runs at full speed on Hopper and Blackwell GPUs. On our Ampere A100s, vLLM falls back to the Marlin kernel. On an H100, throughput would likely be higher, but the cost comparison shifts too (H100 pods start at $1.99/hr).
| Feature | GPT-oss 120B | Qwen3-30B-A3B |
|---|---|---|
| Architecture | MoE (128 experts, 4 active) | MoE (30B total, 3B active) |
| Total parameters | 120B | 30B |
| Active per token | ~8-10B (estimated) | 3B |
| Quantization | mxfp4 (needs Hopper) | GPTQ-Int4 (universal) |
| VRAM required | 76.6 GB x2 GPUs | ~16 GB x1 GPU |
| Max context | 131,072 tokens (128K) | 32,768 tokens (32K) |
| License | MIT | Apache 2.0 |
Production
Higher quality (11/11 vs 10/11), similar throughput on half the hardware, and less than half the hourly cost. For structured output workloads like email classification, JSON extraction, and document processing, the 30B model is the clear choice.
GPT-oss is not a bad model. It is a general-purpose model being tested on specialized tasks. For teams that need:
GPT-oss would be worth re-evaluating on those workload profiles. For our production use case, it does not clear the bar.
GPT-oss has 4x the total parameters of Qwen3. It scored lower on our production benchmark. This is not surprising if you work with these models daily, but it contradicts the default assumption that bigger models are better models.
What matters is task-specific accuracy, structured output precision, and cost per inference. A 30B model that nails your workload every time is more valuable than a 120B model that gets it right 90.9% of the time at 2.3x the cost.
We publish these benchmarks because the model leaderboards do not tell this story. MMLU and HumanEval will not tell you whether a model can correctly classify a contract review email as "action_required" instead of "urgent." Production testing does.
Methodology
11 tests across 6 categories: structured output (JSON extraction from contacts and meeting notes), email classification (urgency, newsletter, action-required), summarization (financial documents), instruction following (format compliance and arithmetic), reasoning (capacity calculation), code generation (Python function), and email drafting. Each test has explicit pass/fail criteria based on output correctness and format compliance.
GPT-oss 120B does not fit on a single A100 80GB GPU. The mxfp4 weights require 76.6 GB VRAM per GPU with tensor parallelism across 2 GPUs. This is not an artificial handicap; it is the minimum viable hardware for this model. Qwen3 at GPTQ-Int4 uses roughly 16 GB on a single GPU. The hardware gap is part of the production cost comparison.
This is one of several model evaluations we publish. For the Qwen3.5 vs Qwen3 comparison and the full story on our self-hosted inference infrastructure, see the research index.
All ResearchBenchmarks conducted February 26, 2026. GPT-oss 120B tested on 2x A100 SXM 80GB with mxfp4 quantization (Marlin backend). Qwen3-30B-A3B-Instruct-2507 tested on 1x A100 PCIe 80GB with GPTQ-Int4 quantization. Quality suite: 11 tests across 6 categories under production conditions. Concurrency tested at 1, 2, 4, and 8 simultaneous streams.
We use cookies to improve your experience. Cookie Policy · Privacy Policy