DeepSeek-V4.1-Flash breaks 1,000 tokens per second for one user: how we roughly doubled decode in five days

On our B300 nodes, DeepSeek-V4.1-Flash, a 1M-token-context open-weight mixture-of-experts model, now breaks 1,000 tokens per second for one user. Tokens are the word pieces the model writes. On our bench, the fastest single-stream request reached 1,064, and 5 of 208 passed 1,000. The average over 14 engine starts, or boots, was 870, about double our first boot's 439, five days earlier, though each step was timed on its own bench. In live traffic on 30 October 2026, our router timed just over half of one lightly loaded node's replies above 1,024. That rate counts short bursts, so short replies read high. The engine's mean gap between tokens on that node works out to 873, in line with the bench.

The biggest step was NVIDIA's low-latency "megakernel" plugin for SGLang, the open-source serving engine, released on 18 October 2026. A megakernel fuses the many small programs the model runs for each token into a few long-running ones on the graphics processor (GPU). NVIDIA built it on four GB300 (Grace Blackwell Ultra) GPUs, where one user's reply streamed 1.38 times faster.

Plain B300 is the same chip family, but it has 148 streaming multiprocessors (SMs), the chip's compute blocks, where GB300 has 152. It also sits in an ordinary x86 server. At launch, the plugin checked for 152 SMs and refused to start on B300. We patched the check and added a safety layer. Decode, the phase where the model writes its reply token by token, now runs 33 to 39% faster for one user than our tuned production stack. Quality held, and it has served production traffic since 30 October.

The short answer

• For one user, the model breaks 1,000 tokens per second: the fastest bench request hit 1,064.
• Live, our router timed just over half of one quiet node's replies above 1,024.
• On average, one stream ran at 870 over 14 bench boots and at 873 live.
• That is about twice our first boot's 439, five days earlier, though the benches differed.
• NVIDIA's megakernel was the biggest step: 33 to 39% faster for one user than our tuned stack.
• The gain shrinks with more users: about 30% at 2 streams, 12 to 15% at 8, 1% at 16.
• Quality held, and 20,818 requests ran with no hang. That caps the risk; it does not rule it out.
• Check that the megakernel really loaded: two setups start cleanly, then run without it.
• It costs a quarter of the cache space and a different split of the model across GPUs.

scroll sideways for the rest

How we got here

The five steps in Figure 1, each timed on that day's bench:

• 26 October, first boot: 439. On our ramp bench, an early copy of the appendix's decode test.
• 27 October, engine tuning: 443 → 516. Block verification of drafted tokens, a fused way to combine results across GPUs and a faster prompt path, on an older bench tool.
• 28–29 October, fast host cores: 514 → 614 to 639. The scheduler pinned to the host's fast cores, on the ramp bench (see the tip below).
• 28–30 October, the megakernel: 638.4 → 885.8 and 849.0. Two boots with our B300 patch against our tuned stack without it, on the megakernel's speed gate.
• 30–31 October, the build we serve: 870.2. Newer engine code merged with no change in speed. Its 14 boots averaged 840.9 to 906.7.

The megakernel step was the largest, and its gain depends on how many users share the node (Figure 2).

scroll sideways for the rest

On the bench, only 5 of 208 single-stream requests passed 1,000, the fastest at 1,063.8 (Figure 4). Live on 30 October, our router timed every reply on both nodes for 34.5 minutes. On our second node, just over half of the replies came in above 1,024 tokens per second; on the first, fewer than half did. The router's rate counts short bursts of accepted drafts, so short replies read high. Two more caveats sit with that share:

• Live prompts suit speculative decoding better. It drafts several tokens ahead and keeps the ones the model agrees with: about 4.0 per step live, against 3.3 on the bench.
• The second node was the quieter one, with 0.25 requests running on average against 0.84 on the first.

What NVIDIA shipped, and why B300 was refused

The plugin is sgl-nvidia-mega, marked experimental, and NVIDIA's team described it in a post with the SGLang team. On four GB300 GPUs replaying agent traces, one user's decode rose from 671.2 to 928.4 tokens per second at the 90th percentile. With 8K-token inputs, the gain was 12 to 38% at 1 to 8 users, at a simulated 3.51 accepted draft tokens per step.

The shipped code required GB300's 152 SMs or stopped at startup. Its page promises "no silent fallback", so refusing B300 was a design choice, not a bug.

On 30 October, upstream merged an official B300 route, pull request #35. Opting in with a setting named sm148 admits a 148-SM B300, and the kernels are unchanged. Upstream measured B300 at one request only: a median of 633 tokens per second without the megakernel and 823 with it, 30% faster.

data (csv)

What we changed

A small patch

Our patch, on the plugin version SGLang pinned on 28 October 2026, changes two settings and none of NVIDIA's kernels:

1. It sets the plugin's resident block count to 148 instead of 152, to match B300's SMs.
2. It declares that the plugin works with host cache offload, SGLang's HiCache, which keeps older cache pages in host memory for our long-context serving.

The second one matters. In an earlier trial with offload off, one stream still gained 15%. But the first token took 3 to 6 times longer, and total output at 4 to 16 streams fell 6 to 12.5%. Our patch predates upstream's route and is unofficial. Upstream's change still lacks the offload flag, so we stay on ours until the next plugin update.

Before you try it: two traps and a tip

• Trap 1, the interface version. The plugin moved to version 2 of its megakernel interface. SGLang code that speaks version 1 still starts with it installed, shows no error and serves at the old speed. Our gate now checks that all 27 megakernel decode graphs were captured.
• Trap 2, the sm148 setting. Upstream's B300 route uses SGLANG_NVIDIA_MEGAKERNEL_SLOTS, which replaces the plugin's 39 default kernel slots instead of adding to them. Set it to sm148 alone, and B300 starts but runs stock code on every step. Keep the default slots in the list, then check that the speed moved.
• A tip for x86 hosts. On one of our hosts, only 16 of 128 cores boost to 4,600 MHz; the rest stay at 2,300. Pinning SGLang's scheduler to the fast cores cut a one-stream step from about 7.1 to 5.4 milliseconds, 20 to 24% more tokens per second. Grace hosts have no such split, and we have not re-measured it with the megakernel on.

The safety layer

A megakernel step spans all four GPUs and keeps them in sync itself. If one GPU drifts, say by sampling a different token, the step can wait forever. A community port to four B200 GPUs hung twice in 12.5 hours, about once every 6 hours, and NVIDIA's post gives no stability figures. So we added four guards before serving real traffic (Figure 3):

• A GPU-agreement probe. Every 100 decode steps, the four GPUs compare a hash of their state, and a mismatch stops the engine on purpose. It costs under about 1% per step.
• Matched random draws. All four GPUs draw the same random numbers, so sampling cannot split them.
• A hang watchdog. If steps stall, it kills the engine, and a supervisor restarts and warms it up.
• Router health checks. A node that misses its two-second check is marked down. When we froze one GPU's process for 9.9 seconds, every router dropped the node within 5.6 seconds, and its three streams paused, then finished normally.

Every engine change also passes a gate before it serves: greedy output compared byte for byte on 32 prompts, NVIDIA's accuracy check, and tool-call format and math under 16 streams. It has already rejected one upstream change that left speed alone but shifted the model's output.

scroll sideways for the rest

Speed

Gain by number of streams

Table 2 gives the numbers behind Figure 2. The baseline is our tuned production stack without the megakernel, on the same node, not stock SGLang.

data (csv)

The megakernel cuts the time each step takes. With few streams, that time is most of the wait; at 16, the batch's own work dominates and the gain is gone. The same new code with the megakernel off matched the old stack on all 19 speed-gate checks, so the gain is the megakernel's.

One stream, across many boots

Speed varies from boot to boot, even on the same code. Across 14 boots on both nodes, one stream averaged 870.2 tokens per second, with boot averages from 840.9 to 906.7, about 8% apart. Our two code versions ran the same, at 875.6 and 863.0. So we judge a change by many boots, or against the normal spread between boots.

Figure 4 sets the megakernel against stock kernels: the same new code with the megakernel switched off, on the tuned stack's own expert layout, and on the same bench. The megakernel ran 38 to 44% faster, and even its slowest request beat the fastest stock one. That tops Table 2's 33 to 39% because this stock boot averaged 615.3 against the old stack's 638.4, a gap inside the normal spread between boots.

scroll sideways for the rest

Live traffic agrees with the bench. Over the 34.5-minute window above, the engine's mean gap between tokens on our quieter second node was 1.145 milliseconds, or 873 tokens per second, beside the bench's 870. On the busier first node, with more streams at once, it works out to 702.

First token and long prompts

On an idle node, a short prompt's first token arrived in a median 39 milliseconds, and in 26.8 on our faster host. Long prompts load fast too. On our faster host, a never-seen 60,000-token prompt filled in 0.869 seconds while four other streams kept decoding. A 979,971-token prompt, near the 1M-token context limit, filled in 15.4 seconds, and the model found the facts hidden in it. That speed comes from our own prompt-path changes; the megakernel left prompt filling unchanged.

Quality

Against our tuned stack on the same tests and seeds, nothing got worse in 9 paired tests, from grade-school math to multi-step agent sessions. Against our reference scores for the same weights on an 8×B200 node, every row of Table 3 is within noise.

data (csv)

NVIDIA's own accuracy check re-scores the output and measures how far the token odds moved. It passed on every boot we checked, and the worst reading was about a quarter of its limit. Most of our accuracy sets have 100 to 300 items, so they would miss a drop smaller than about 7 to 15 points.

Stability

Two one-hour soak tests, with the probe checking every 10 steps, served 13,410 requests with no hang and no mismatch in 852,300 probe checks per GPU. Production on 30 October added 7,408 requests with no hang, and one engine ran 13 hours without a restart. The soak and that build's first production hours served 18,872 requests in 2.095 loaded hours, counting only time with a request running.

Zero hangs over those 18,872 requests caps the rate at 1.43 per loaded hour (95% confidence). That is a bound, not proof of zero, and it cannot yet rule out the community port's one hang every 6 hours; that takes about 18 hang-free loaded hours. A stopped node rejoins the pool in about 14 minutes with no hands-on step.

What it costs

The plugin needs every GPU to hold a slice of every expert, the small networks a mixture-of-experts model picks from for each token. Our tuned stack gave each GPU whole experts instead. That layout shrinks the cache that holds each conversation's memory by 25%, still far more than one 1M-token request needs. It also nudges output: fed the same text, the build picks the old stack's top token at 94.3% of short-prompt positions. A plain engine with that layout flips the same tokens, and no accuracy test we ran could see it. With the node kept busy, the one-stream speedup cuts cost per output token by 25 to 28%.

What we do not claim

• Not every reply passes 1,000 tokens per second. The bench averages 870 and live traffic 873; only 5 of 208 bench requests passed 1,000, and just over half of one quiet node's live replies topped 1,024.
• Not an exact doubling. The steps used different benches, so 439 to 870 is about 2×.
• Not faster than GB300. Our 90th percentile, 954.0, sits near GB300's published 928.4 and 945.96, but the workloads, weights, draft acceptance and statistics all differ. Our fastest request, 1,064, does not compare either. What carries over is the relative gain, on hardware the code was not written for.
• Not a supported setup. Our patch is unofficial and needs re-checking at each plugin update; upstream's sm148 opt-in is the supported route. NVIDIA's code is not broken on B300; it refused B300 on purpose.
• Not every workload. Speeds include speculative decoding and come from short synthetic prompts, 300-token replies and an idle node. Production peaked at 3 to 6 requests at once, and soaks sampled at temperature 1.0 only.

Appendix: method

Hardware. Two rented nodes, each with four B300 SXM6 GPUs (148 SMs, compute capability 10.3, 275,040 MiB each, NVLink between every pair, 1,100 W, 2,032 MHz SM clock) on x86 hosts.

Serving stack. Our SGLang fork at tensor parallel size 4 and expert parallel size 1, FlashMLA attention, an fp8 (e4m3) KV cache, MXFP4 expert weights and 5-token draft blocks with block verification. Context 1,048,576 tokens, chunked prefill 8,192, up to 64 running requests, decode graphs for batches of 1 to 64. HiCache host offload at ratio 2 with selective write-through, which needs our patch's offload flag; TensorRT-LLM all-reduce fusion; the plugin enabled with --enable-nvidia-megakernel. The baseline is the same fork tuned without the megakernel, at expert parallel size 4. The TP-consistency probe runs every 100 decode steps in serving and every 10 in soaks, with assert on.

Decode speed. Four synthetic reasoning prompt templates with random numbers, tens of tokens long; thinking on, temperature 1.0, top-p 1.0, 300 completion tokens per request. At concurrency C, C streams start together; each cell has 8 repetitions, seeded per C. Per request, decode speed is completion tokens over the time from first to last streamed token, so time to first token is excluded. A repetition's value is the median over its C streams; a cell is the mean over repetitions. Every node was out of the router pool and idle before each run. Accepted draft length is about 3.3 per step here and about 4.0 in live traffic.

Speed gates. Each gate sets a band from the baseline's own boot-to-boot (A/A) spread: about ±4.6 to 8.3% for decode, 8 to 32% for time to first token and 6.7% for cold fill. A cell outside the band is a gain or a regression; a gate passes with no regression. One boot per arm understates boot variance, which is why the single-stream figures use 14 boots.

Speed history. Figure 1's steps come from four benches, as logged at the time: the ramp bench (first boot and fast host cores), an older command-line tool (27 October), the megakernel speed gate (28–30 October, and Figure 4's three gate boots) and the decode test above (the 14 boots). The ramp bench is an earlier copy of the program the decode test wraps, and we do not hold its first-boot settings, so steps on different benches do not compare as numbers.

Live speed. Over 34.5 minutes on both nodes on 30 October 2026, our router timed each reply's decode rate over its visible stream and counted the rates in histogram buckets. Two of its percentiles for the second node fall inside the 1,024 to 2,048 bucket, where it reads them by straight-line interpolation; together they put 51.5% of that node's replies above 1,024. The rate includes bursts of accepted drafts, so it is not a sustained per-stream speed. That speed is the engine's mean gap between tokens over the same window: 1.145 milliseconds on the second node, 873 tokens per second, and 1.424 on the first, 702. The engine's percentile gaps are not used, since they fall inside its first histogram bucket.

First token and cold fill. Time to first token is client-side, from send to the first streamed delta, on an idle, warm node. Cold fill starts four background decode streams, then sends one never-seen 60,000-token prompt, 8 repetitions; fill speed is its length over time to first token. The 1M-token probe is one 979,971-token prompt with hidden facts, checked for the answer and the time.

Numerical checks. Greedy: 32 prompts at temperature 0, 256 tokens, compared byte for byte on an idle node, since batch composition changes decoding. Teacher-forced: the reference's output is fed to the candidate over 32 prompts and top-1 agreement recorded; the long-prompt half sits at a 94 to 95.5% noise floor, so the short half carries the signal. NVIDIA's KL test: temperature 0, the k3 estimator on the sampled token, re-scored by a prefill pass, the mean of per-prompt means; short set 8 LongBench-v2 prompts × 256 tokens, limit 0.03; long set 4 windows of 18,000 to 27,000 tokens × 128 tokens, limit 0.06; 3 repeats per boot. Readings ran 0.0034 to 0.0076 short and 0.0087 to 0.0162 long.

Accuracy. Newcombe 95% intervals for differences in proportions, McNemar's test for paired items. A build is non-inferior when the interval's lower end is above −3 points. The run against the 8×B200 reference used concurrency 4, temperature 1.0 and two repetitions pooled, unpaired.

Hang bound. With zero events in T loaded hours, the one-sided 95% upper bound on a Poisson rate is 2.996 / T. Builds are never pooled; a build's soak and serving hours are. The later build that took over production on 30 October bounds at 8.76 on its own 0.342 loaded hours.

Data handling. Every speed and quality prompt is synthetic or from a public benchmark. Live traffic was read only as aggregate metrics.

Sources

• SGLang, NVIDIA and AMD teams, SGLang low-latency mode: kernels and runtime, 18 October 2026.
• The plugin's code and requirements: sgl-project/sgl-nvidia-mega.
• Upstream's B300 opt-in and its one-request measurement: sgl-nvidia-mega pull request #35, merged 30 October 2026.
• SGLang's pin to the tested plugin version and its GB300 screen: SGLang pull request #47145, merged 28 October 2026.

@techreport{ezolabs2026megakernelb300,
  title       = {DeepSeek-V4.1-Flash breaks 1,000 tokens per second for one user: how we roughly doubled decode in five days},
  author      = {{EZO Labs Research}},
  institution = {EZO Labs Inc},
  type        = {Research report},
  number      = {EZO-R-2026-07},
  year        = {2026},
  month       = oct,
  url         = {https://ezolabs.io/research/megakernel-b300}
}