Applied AI
Benchmarking four approaches against real lab data: handwritten notebook scans, patents, legacy protocols. The cheapest model failed completely. The most expensive tied with the second-cheapest. The real bottleneck was neither.
July 2026 | Dry Ground AI Research
The Problem
A biotech client needed to backfill structured reaction data across their chemical catalog. The source material was messy by nature: handwritten notebook scans, PDFs of patents, legacy protocols, published literature. The goal wasn't summaries. They needed precise, structured stoichiometry: reagents, equivalents, limiting substrates, yields.
Of their 5,570 chemicals, 1,364 had at least one procedure document to work from. Most had 1 to 3 docs. A small tail had up to 12. We needed to find the fastest, most accurate, and most cost-effective path to process all of them.
We benchmarked four approaches against a validation set of 9 chemicals, selected to represent the range of document types and complexity. Claude Opus 4.8 served as the quality baseline.
Benchmark Results
The results broke cleanly into three buckets. One model disqualified itself. Two models tied on quality. One model opened a different architecture entirely.
| Model | Reads scans | Reagent IDs | Stoichiometry | Speed | Verdict |
|---|---|---|---|---|---|
| Claude Opus 4.8 | Correct | 30-90s | Gold standard | ||
| Claude Sonnet 4.6 | Matches Opus | Slower* | Best quality/cost | ||
| Claude Haiku 4.5 | Blank output | ~75s | Rejected | ||
| Gemma 4 26B | Noisy numbers | Fast (server fetch) | Hybrid OCR only |
* Sonnet's timeouts were caused by a CLI timeout cap set to 5 minutes (since raised to 10). Not a quality issue.
Finding 1
Four of nine validation chemicals came back from Haiku with zero extracted steps. It couldn't read the handwritten notebook scans. That's not a tunable problem -- it's a hard capability floor. For a catalog where scan-based documents make up a significant portion of source material, Haiku is out at step one.
The cost advantage is real: Haiku runs at roughly 15% of Opus's API cost. But a 15% price tag on an extraction that produces nothing isn't a discount. The other three models could all read the scans. Haiku couldn't. That ended the conversation.
Finding 2
On every metric that mattered for this workload (reagent identification, equivalents, limiting substrate, yield extraction) Sonnet 4.6 and Opus 4.8 produced effectively identical output. Not close. Identical.
The reason is worth understanding. Document vision tokens dominate the cost on this kind of extraction. The model spends most of its compute reading the scans. Once it has the image context, the structured reasoning step is not especially hard. Sonnet handles that reasoning just as well as Opus does.
For this workload, Opus doesn't buy anything Sonnet doesn't already deliver. That's not a knock on Opus. It's a signal that you should benchmark your actual task before defaulting to the most capable model.
Finding 3
Gemma 4 26B runs on our self-hosted RunPod infrastructure at no marginal API cost. It reads the handwritten scans Haiku refused. But its stoichiometric numbers aren't reliable. Initial runs reported raw amounts as equivalents (a systematic prompt issue we fixed), but even after correction, quantitative values drifted run to run. The 26B parameter count isn't enough for consistent quantitative reasoning on this task.
Gemma alone isn't the answer. But here's what it suggested: Gemma doesn't need to do the chemistry reasoning. It just needs to transcribe.
A 26B model can OCR a handwritten scan accurately. Claude can structure clean text without processing images at all. When Gemma transcribes first and Sonnet reasons on the text output, input tokens drop 50 to 100 times compared to vision-direct processing. The reasoning pass is fast and cheap. On the hardest multi-page cases, where procedure values conflicted across documents, the hybrid recovered Opus-level equivalents.
The cost difference is significant. Pure Sonnet vision across all 1,364 chemicals runs roughly $390 on Batches. The hybrid -- Gemma OCR at $0 plus Sonnet text-only reasoning -- comes to around $55 total. Same output quality. 86% cheaper.
Gemma 4 26B fetches documents server-side and transcribes them to text. Sonnet receives clean text (no images) and outputs structured stoichiometry. Vision tokens drop out of the reasoning pass entirely.
One result worth noting: substituting Opus for Sonnet as the reasoner produced no quality gain. On clean transcribed text, both models get the same answers. The ceiling in this architecture is Gemma's transcription fidelity. Upgrading the reasoner doesn't raise it.
Total cost for all 1,364 chemicals: ~$55. Compared to ~$390 for pure Sonnet Batches or ~$649 for pure Opus Batches. The 91% cost reduction comes entirely from shifting vision work to a self-hosted model.
Finding 4
Cost wasn't the constraint. Neither was compute. Running the pipeline locally, a 2.45 MB scan took 160 seconds to download from S3 -- roughly 15 KB/s. General internet on the same machine was running at nearly 4 MB/s. That's a 300x slowdown specific to the S3 route.
With per-chemical processing averaging 3 to 4 minutes locally, throughput settled at about 15 chemicals per hour. Two days of wall time to finish 595 chemicals. The model was sitting idle most of that time.
The fix is running the pipeline from an EC2 instance in the same AWS region as the S3 bucket. In-region bandwidth runs around 100 MB/s. That drops wall time to 12 to 18 hours. The hybrid path sidesteps this entirely -- Gemma fetches images server-side, directly from storage, without touching the slow local connection.
A 2.45 MB scan: 160 seconds locally vs an expected sub-second transfer in-region. If your extraction pipeline feels slow, check your data access layer before you look at model latency.
Cost breakdown
For a backfill at this scale, cost was never the deciding factor. Even Opus at full API price is a one-time project cost under $1,300. The question was always quality. Quality was settled by Haiku's failure and the Sonnet/Opus tie.
| Approach | Opus 4.8 | Sonnet 4.6 | Haiku 4.5 |
|---|---|---|---|
| Opus 4.8 via API (full 1,364) | $1,298 | ~$780 | $188 |
| Opus 4.8 via Batches (-50%) | $649 | ~$390 | $94 |
| Gemma + Sonnet hybrid (full 1,364) | N/A | ~$55 | N/A |
API rates: Opus $5/$25 per MTok in/out. Sonnet $3/$15. Haiku $1/$5. Gemma $0 (self-hosted RunPod). Hybrid Sonnet pass estimated ~$0.04/chem text-only. Vision tokens drive cost on pure-API paths.
Completion paths
| Path | Quality | Est. wall time | Notes |
|---|---|---|---|
| Sonnet via CLI (local) | Best (non-hybrid) | ~3-4 days | S3 download dominates; ~15 chems/hr |
| Sonnet from in-region EC2 | Best | ~1-1.5 days | In-region S3 removes download bottleneck |
| Gemma alone | Low (bad numbers) | ~4-8 hours | No S3; unreliable stoichiometry |
| Gemma + Sonnet hybrid | Near-Opus, noisy equiv | ~6-10 hours | Gemma OCR (no S3) + fast Sonnet text; ~$55 total |
| API via Batches (Opus or Sonnet) | Best | ~2 hours compute | ~$390 (Sonnet) / ~$649 (Opus) |
Recommendation
Quality matches Opus at 40% less cost. No tradeoff on this workload.
The 300x S3 slowdown disappears. Wall time drops from ~2 days to ~12-18 hours.
~$55 total for all 1,364 chemicals. Free vision via self-hosted Gemma, cheap Sonnet text-only reasoning. Chemist review of equivalents catches transcription noise. 91% cheaper than pure Opus Batches.
"On document-heavy extraction work, model cost and reasoning quality are rarely where you find the constraint. We found a 300x performance gap in the data access layer before we ever had to seriously debate model selection. That's usually how it goes."
Dry Ground AI Research, July 2026
We use cookies to improve your experience. Cookie Policy · Privacy Policy