How a model upgrade “got slower,” and what fixing it taught us about serving hybrid LLMs on Blackwell GPUs.
We recently swapped the vision-language model behind an internal batch service to Qwen3.8-27B — a new multimodal model, running 4-bit quantized on a single NVIDIA Blackwell GPU via vLLM. The model was better. The workload got dramatically slower: a representative batch job that used to finish in about 2.5 minutes now took 18 minutes.
This is the story of chasing that 7× regression. The punchline up front: almost none of it was the model’s fault. The upgrade exposed a bug that had been hiding in plain sight for months, and once we understood why, we ended up faster than we’d ever been — with the final job landing at ~3.4 minutes.
Here’s what we learned, in the order we learned it.
The setup
- Model: Qwen3.8-27B, a multimodal (text + vision) model with a hybrid attention architecture.
- Quantization: NVFP4 — a 4-bit format with native kernel support on Blackwell. Weights come to ~24 GB, which leaves comfortable room on a 48 GB card for the KV cache.
- Serving: vLLM 0.23.0, one GPU, FP8 KV cache.
- Workload: a batch service that issues a long stream of short requests — each one an image plus a small prompt, producing ~100 tokens of structured JSON out. A single job fires roughly 130 of these.
Nothing exotic. And that’s the point — the lessons here apply to anyone running a batched, request-heavy workload against a modern inference server.
First surprise: single-stream decoding was slow
The first thing we measured was raw token generation speed with one request in flight. On a 27B 4-bit model on a Blackwell card, you might expect 60–100+ tokens/sec. We saw ~22 tokens/sec.
Two things conspire here, and both are worth knowing about:
1. The architecture. Qwen3.8 is a hybrid model — it mixes standard attention with a linear-attention component (Gated DeltaNet). On current GPUs, that linear-attention path runs through just-in-time-compiled Triton kernels rather than the hand-tuned CUDA kernels that regular transformer attention enjoys. On the newest Blackwell silicon, that kernel gap is real and measurable. It’s improving with every vLLM release, but today it’s a tax you pay.
2. Eager mode. We run with CUDA graphs disabled (--enforce-eager). CUDA graphs capture the sequence of GPU operations once and replay them cheaply; without them, every decode step pays full kernel-launch and Python-dispatch overhead. We disable them because graph capture plus compilation spikes host RAM, and this box is memory-constrained. The cost: single-stream decoding — where there’s no batching to hide that per-step overhead — is exactly the worst case for eager mode.
Keep that 22 tok/s number in mind. It’s the villain of the story, but not for the reason you’d think.
The real culprit: a “batch” function that didn’t batch
We instrumented a full job and let it print a per-phase timing breakdown. One phase dominated everything: the vision-inference step was 88% of the total wall-clock — 966 seconds out of 1092.
Watching the inference server’s live metrics during that phase told the whole story in one number: Running: 1. For the entire 966 seconds, the server never had more than a single request in flight.
The client code looked like this (simplified):
def read_crops_batch(url, crops, prompt):
"""Call the model for each crop. Returns one result per crop."""
return [read_one(url, crop, prompt) for crop in crops]It’s called a batch function. There’s even a batch_size setting in the configuration. But look at what it does: a plain list comprehension that fires one HTTP request, blocks until it returns, then fires the next. It was strictly sequential. ~130 requests, one at a time, each waiting ~5 seconds. 130 × 5s ≈ your afternoon.
Why did the upgrade “cause” this?
Here’s the crucial insight, and it’s a general one:
Serialization multiplies per-request latency. The old model was fast enough per request to hide the multiplier.
The previous model was a conventional dense transformer with mature CUDA kernels and CUDA graphs enabled — about 0.4 seconds per request. Run 130 of those sequentially and you’re done in under a minute. Nobody noticed the loop was serial, because at 0.4s a serial loop is fine.
The new hybrid model, with its Triton kernels and eager mode, runs ~4–5 seconds per request. Same loop, same code, 10× the per-request cost:
| Old (dense) | New (hybrid) | |
|---|---|---|
| Per request | ~0.4 s | ~4–5 s |
| ~130 requests, serial | ~1 min | ~10+ min |
The upgrade didn’t create the bottleneck. It exposed a latent one. The serial loop had been quietly wasting the GPU for as long as it existed — the old model was just quick enough that the waste never showed up on anyone’s stopwatch.
The lesson: when a system gets slower after a component change, the change is often the messenger, not the cause. Look for the thing that the old component’s speed was papering over.
Fix #1: Actually batch (and measure how far to push it)
The fix is embarrassingly small — wrap the loop in a thread pool so requests fly concurrently and let the inference server do what it’s built for (continuous batching):
def read_crops_batch(url, crops, prompt, workers=12):
with ThreadPoolExecutor(max_workers=workers) as pool:
return list(pool.map(lambda c: read_one(url, c, prompt), crops))pool.map preserves order, so the output is byte-for-byte identical — this is purely a “stop wasting the GPU” change, not a behavioral one.
But how many workers? Rather than guess, we ran a concurrency sweep — the same fixed set of requests at increasing parallelism — and measured aggregate throughput:
| Concurrency | Throughput | Speedup vs. serial |
|---|---|---|
| 1 | 21 tok/s | 1.0× |
| 4 | 80 tok/s | 3.7× |
| 6 | 116 tok/s | 5.4× |
| 8 | 154 tok/s | 7.2× |
| 12 | 210 tok/s | 9.9× |
| 16 | 266 tok/s | 12.5× |
Two things jump out:
- There’s no knee. Throughput keeps climbing near-linearly all the way to 16. We didn’t hit a hardware ceiling; we hit the server’s configured concurrency cap (
max_num_seqs=16). - Eager mode is why batching pays so well here. Because single-stream leaves the GPU so underused, every additional concurrent request finds idle compute to soak up. The same batching would help a CUDA-graph-optimized dense model too, just less dramatically.
We settled on 12 concurrent rather than maxing out at 16 — deliberately leaving headroom so other, latency-sensitive traffic sharing the same server doesn’t get stuck behind a batch job (more on that below).
That one change took the job from 18 minutes to ~5 minutes. No server restart, no model change, no quantization change. Just a thread pool.
Fix #2: Speculative decoding (MTP), for free tokens
With the workload now properly parallel, we reached for a second lever: Multi-Token Prediction (MTP), a form of speculative decoding.
The idea behind speculative decoding: a small, cheap “draft” predicts the next token (or few), and the big model verifies several candidates in a single forward pass instead of generating them one at a time. When the draft is right, you get multiple tokens for the price of one step. When it’s wrong, you fall back — but crucially, the output is mathematically identical to not using it at all. It’s a pure speedup, not a quality trade-off.
Qwen3.8 ships with a built-in MTP head, and even the 4-bit checkpoint we used included those weights (quantized checkpoints often drop them, so this is worth checking). Enabling it in vLLM is one flag:
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'We re-ran the identical sweep with MTP on:
| Concurrency | Baseline | With MTP | Gain |
|---|---|---|---|
| 1 | 21 tok/s | 31 | +44% |
| 6 | 116 | 163 | +40% |
| 8 | 154 | 217 | +41% |
| 12 | 210 | 280 | +34% |
| 16 | 266 | 279 | +5% |
A consistent +34–44% across the board — and the reason it works so well here is telling. The model’s draft acceptance rate was 84% (and 85–97% on real traffic). That’s unusually high, and it’s because our outputs are short, highly-structured JSON. Predictable text is easy to draft; the speculative token is almost always right. If you’re generating boilerplate, structured data, or formatted output, speculative decoding is a much bigger win than it is for open-ended prose.
Two nuances worth flagging:
- The gain collapses at the highest concurrency (+5% at 16). At that point the GPU is closer to compute-bound, and drafting competes with batching for the same compute. Speculative decoding gives its biggest wins when there’s spare compute — which, thanks to eager mode, we have plenty of at our operating point of 12.
- It costs a little memory. The draft head adds ~0.8 GB, and the KV cache shrank to make room. Fine for short requests; something to watch if your contexts are long.
MTP pushed the job to ~3.4 minutes.
Bonus lesson: know how your requests share the server
Once a single job can fire a dozen concurrent requests, a natural question arises: does one batch job hog the whole server? Does other traffic have to wait?
Understanding this is worth a few minutes, because the answer shapes how you tune concurrency. In our case there are two independent queue levels:
- The job scheduler runs one batch job at a time. A second submitted job waits in that scheduler’s queue — it never even reaches the inference server until the first finishes. So two batch jobs never fight over the GPU.
- The inference server has a fixed number of slots (16, in our config). A single job uses at most 12 of them. That leaves 4 slots always free for other, interactive traffic hitting the same server — which runs immediately, without queuing, as long as it doesn’t itself need more than the free slots at that instant.
This is why we chose 12 workers instead of 16. Maxing out would make each batch job marginally faster while forcing interactive requests to queue behind it. Twelve is the balance point: batch jobs run fast, interactive stays responsive. If your server is dedicated to one workload, max it out; if it’s shared, leave headroom deliberately.
The scoreboard
| Time | What changed | |
|---|---|---|
| Original (after model swap) | 18.2 min | serial client, one request at a time |
| Parallelized (12 workers) | ~5.3 min | thread pool — client-side only |
| + MTP speculative decoding | ~3.4 min | one server flag, lossless |
The dominant phase — model inference — went from 966 seconds to ~97 seconds, a 10× improvement, with output identical at every step.
Takeaways
If you’re deploying a modern LLM — especially a hybrid-architecture or heavily-quantized one on new silicon — here’s what this exercise reinforced:
- A regression after an upgrade is often a messenger. The new component didn’t break anything; it removed the slack that was hiding an existing problem. Measure per-request cost and how requests are dispatched.
- “Batch” in a function name is not a promise. Verify that your client actually issues concurrent requests. A sequential loop against a continuous-batching server wastes almost all of its capacity — and the waste is invisible until per-request latency rises.
- Sweep concurrency; don’t guess. We assumed the sweet spot would be ~6–8. The data showed throughput scaling to the server’s cap with no knee. Ten minutes of measurement beat an afternoon of intuition.
- Speculative decoding is close to free — and shines on structured output. It’s lossless, it’s one flag, and if your model generates predictable text, acceptance rates (and speedups) are high. Check whether your checkpoint actually includes the draft weights.
- Understand slot-sharing before you tune. How your batch and interactive traffic coexist on one server determines whether “max concurrency” is the right answer or the wrong one.
The biggest win of the whole effort cost three lines of code and no downtime. The most satisfying part was realizing the “slow new model” was never slow — we’d just been asking it to work with one hand tied behind its back.