
Optimizing Inkling 2 Small's kernels on NVIDIA B300
, 11 min read, By Jaber Jaber
ResearchRightNow AI is a YC-backed GPU research lab. We optimized the kernels of Inkling 2 Small, Thinking Machines' 594B parameter mixture of experts, on 8 NVIDIA B300 GPUs. Every kernel, test, config and result is open source at github.com/RightNow-AI/inkling-b300-kernels.
Thinking Machines pretrained, midtrained and post-trained this model in NVFP4, and their model card calls it "the native precision for the model". We run those numbers as they were trained: the NVFP4 experts of layers 0 to 39 on Blackwell's block-scaled FP4 tensor cores, the MXFP8 experts of layers 40 to 47 on its block-scaled FP8 ones.
| Measured on B300 GPUs, 2026-11-21 | Result |
|---|---|
| Same logits, 64 rows per step against 1 row per step, full model on 8 GPUs | 0 of 599 rows differ, bit for bit |
| Next-token log-probability difference from vLLM, same model and GPUs | 0.025 nats mean, 0.13 at p95 |
| w13 expert GEMM, one GPU, one token, rebuilt against our first kernel | 5.1x faster on NVFP4, 7.5x on MXFP8, same bits |
| Kernel test suites on one B300, 8 ranks emulated | 10 of 10 pass |
Every kernel here was written by our AI kernel engine: coding agents that write CUDA, run our test harness and keep or revert each change. A kernel was kept only after it passed its gates.
Why this needs CUDA
NVFP4 stores 4 bit E2M1 values with an E4M3 scale for every block of 16. Blackwell's tensor cores read those scales directly: one tcgen05.mma instruction, kind::mxf4nvf4.block_scale.block16, multiplies the model's own bytes.
AMD's MI350X and MI355X do not. In AMD's own words, their FP4 compute support is limited to MXFP4, whose blocks are 32 wide with power of two scales, so an NVFP4 checkpoint must first be converted to MXFP4 to use that path. On Blackwell the model runs in the numbers it was trained in.
Same bits at 1 or 64 rows
Most inference engines are close, not exact. A token's result depends on what else shares its step, because the batch changes how sums are split and floating point addition is not associative.
Thinking Machines named the fix batch invariance: a row's result may not depend on the batch at all. We keep it through NVFP4 and MXFP8 block-scaled tensor cores, experts split across 8 GPUs and every exchange between them.
Every reduction has a fixed shape. Sums run in fixed chunks and balanced trees, sums across GPUs fold in rank order in FP32, and the build turns off automatic contraction, so every fused multiply-add is one we wrote on purpose.
Every property has a broken twin. Each bit-identity property is paired with a deliberately broken variant that must be caught. Where a break cannot move the bits, such as reordering sums that are exact, the log says so instead of counting it.
The proof runs on the real model. Feeding each prompt 64 positions per step and 1 position per step gives identical logits on all 599 rows we compared. Every cell below is one row: hover or tab through it to see both hashes.
Logit hash per row, 64 rows against 1 row per step
599 of 599same bits
essay143 of 143 same
code152 of 152 same
math157 of 157 same
explain147 of 147 same
essay, row 0same bits
- 64 rows a step
- 61984ec4553e0829
- 1 row a step
- 61984ec4553e0829
Every chunked sum in our expert GEMMs ends in this balanced tree; each add is an explicit round-to-nearest add that the compiler never fuses into a multiply-add, and the swap at the top is the broken twin our tests must catch.
Kernel 1: NVFP4 experts split across GPUs, exactly
The activations of an NVFP4 expert are quantized at run time with an FP32 scale per token, computed from the maximum over the whole row. SGLang's recipe for this model says the NVFP4 experts cannot be TP-sharded, "because their activation scales must cover each expert's full input width", so engines run them with expert parallelism, whole experts per GPU.
We sharded them anyway: one eighth of every expert's columns per GPU, and an exchange of the row maximum. A maximum does not depend on the order it is taken in, so the exchange is exact and the quantizer matches the reference's four over six rule bit for bit.

Each GPU takes the largest magnitude in its own 288 columns of the row, and every GPU then folds the 8 maxima it receives with fmaxf, which gives the same bits in any order; the fma line is the order-dependent twin our permutation test must catch.
The exchange gives the same bits in all 8 rank orders we tested, with the 8 ranks emulated on one B300, and an order-dependent combine is caught.
Every GPU reads an equal slice of every active expert, so no GPU reads more expert bytes than the others. The price is one more exchange per FP4 layer.
A second form exchanges the activation itself instead of its maximum, and splits w2 by output rows instead of by K.
Every GPU then quantizes the full row on its own, the closing all-reduce of FP32 partials becomes an all-gather of BF16 slices, and each GPU sends 3.2 times fewer bytes per FP4 layer: 40 KB against 129 KB at one row, 0.32 MB against 1.03 MB at an 8 row verify step.
It rounds where SGLang rounds, each expert's output to BF16 before the sum, and shows zero error against an FP64 model of those rounding points on the FP4 layer we tested.
Kernel 2: the block-scaled GEMM, rebuilt for one token
Both kernels run tcgen05.mma with the weights as M, 128 rows per tile, and the tokens swapped into N, with FP32 accumulators and the block scales in tensor memory.
In the rebuilt kernel a single thread feeds the tensor core: for each K step it copies the weight and token block scales into TMEM with tcgen05.cp, then issues one block-scaled tcgen05.mma whose accumulate predicate is off on the chunk's first K step, so every chunk starts from a zeroed FP32 accumulator.
At one token, the w13 GEMM on each GPU streams 11.9 MB of FP4 weights and scales per layer. Our first kernel gave each expert tile one CTA and walked K in stages it never overlapped: 40 CTAs on 148 SMs, at 114 GB/s.
The rebuilt kernel splits the fixed K chunks across CTAs and pipelines every stage's loads under the MMAs. Each chunk is still the same chain of block-scaled MMAs in the same K order from a zeroed accumulator, and the last CTA to arrive combines the chunks in the same fixed tree, so every output keeps its bits.
Each CTA writes its chunk's FP32 partials, fences and takes a ticket; the last one to arrive resets the counter for the next CUDA graph replay, reads every chunk through L2 in chunk order and sums them in the same fixed tree.
Microseconds per launch
- w13 · NVFP45.1xFirst kernel 104.4 microseconds, rebuilt 20.5.
- w13 · MXFP87.5xFirst kernel 184.3 microseconds, rebuilt 24.6.
- w2, first form · NVFP41.0xFirst kernel 14.3 microseconds, rebuilt 14.3.
- w2, first form · MXFP81.3xFirst kernel 20.5 microseconds, rebuilt 16.3.
- w2, second form · NVFP43.3xFirst kernel 55.8 microseconds, rebuilt 16.9.
- w2, second form · MXFP84.6xFirst kernel 88.0 microseconds, rebuilt 19.1.
w13, NVFP4, layer 0
- first kernel
- 104.4
- 114 GB/s
- rebuilt
- 20.5
- 583 GB/s
- weights
- 11.94 MB
- 8 experts
At 8 tokens per step, with 40 distinct experts per layer, the rebuilt w13 reads 1.39 TB/s on NVFP4 and 1.84 TB/s on MXFP8, against the 6.85 TB/s we measured for a B300's memory, so this kernel still has headroom.
Against the first kernel, on layers 0 and 40 at 1 to 64 rows and over three CUDA graph replays, every intermediate is bit for bit equal: 0 of 512 rows differ.
Kernel 3: a depth mix in one round
Inkling mixes the outputs of earlier layers before every attention and every expert block: a softmax over up to nine earlier outputs, then a normalization. Done directly, each mix needs two reductions across the GPU.
We keep the Gram matrix of the earlier outputs, computed once when each block closes, so the mix's norm comes from that matrix and one new column: one reduction instead of two, 2 rounds per layer instead of 4, counted.
The norm lands within 1.6e-7 relative of the direct one in our tests, with a compensated sum and a fallback to the direct second round for any row where cancellation could exceed the gate.
Folding the expert router into the same reduction changed the top 8 experts of 1.1% and 1.6% of tokens on the real routers of layers 0 and 40, so it stays out: a change that moves routing is not lossless.
How close to vLLM
The full model on 8 GPUs, teacher-forced on vLLM's own continuations of 4 prompts, against vLLM's log-probabilities at 595 positions. vLLM run twice agrees with itself exactly, so every difference below is between the two engines' arithmetic, not noise.
Distance from vLLM, nats
- mean
- 0.0249
- median
- 0.0016
- p95
- 0.1285
- max
- 0.7269
- vLLM run twice
- 0.0000
Where the step goes today
Our engine still runs a decode step as separate launches under CUDA graphs: about 16 ms per step at one row without the draft, on 8 B300 GPUs. The largest cost is the expert GEMM of our first kernel, which the rebuilt kernel cuts about fivefold in isolation but which is not in the engine yet.
Kernel time per step, one GPU, ms
16.76 in all
- Expert GEMMs5.94
- Exchanges and waits4.93
- Depth mix and conv2.45
- Attention1.39
- Router and work list0.88
- Dense BF16 GEMMs0.63
- Quantize and combine0.48
- Embed and sample0.03
Expert GEMMs35.4%
- 4.03w13 GEMM, NVFP4expert_gemm_tc05_kernel<0,1,4,0>
- 1.42w13 GEMM, MXFP8expert_gemm_tc05_kernel<1,1,4,0>
- 0.36w2 GEMM, NVFP4expert_gemm_tc05_kernel<0,1,1,1>
- 0.12w2 GEMM, MXFP8expert_gemm_tc05_kernel<1,1,1,1>
The next piece is one persistent decode program per GPU: every layer in one resident kernel, with the exchanges inside it.
Our byte and latency budget for one verify step reads about 7.8 GB per GPU across 154 exchanges, with Inkling's own speculative draft at 3.35 tokens per step and an assumed 45 distinct experts per layer, which we have not measured yet.
On that budget the program targets about 1,900 tokens per second per user, a projection with a range of about 1,630 to 2,160. For scale, vLLM with the same draft runs this model at 666 tokens per second per user on our own 8 B300 node. The next post reports our measured number, bit for bit.
The configs
The full source, the test suites and the cloud runner that produced every result are at github.com/RightNow-AI/inkling-b300-kernels, under the Apache 2.0 license.
Every kernel is built for the B300's architecture-specific target, with contraction off:
The full model on 8 GPUs, teacher-forced at 64 rows per step and at 1, and the speed line:
The reference:
Conditions
Our engine and the vLLM reference ran on one cloud node of 8 NVIDIA B300 SXM6 GPUs, driver 620.70.06, with peer to peer access between all 8 GPUs; the kernel suites and GEMM timings ran on one of its GPUs.
Checkpoint thinkingmachines/Inkling-2-Small at revision 885d78bf. Reference: vLLM nightly image sha256:c17e9d88. vLLM's speed with the draft (2026-11-21) and the 6.85 TB/s memory bandwidth (2026-11-20) were measured on our own 8 B300 node; everything else on 2026-11-21.