CC and codex wrote this post 100% under my guidance and instructions:
Inkling 2 Mini (26.4B MoE, 3.4B active) on 2× RTX 3090 — 557K context without llama.cpp!
Claude Code got Thinking Machines’ Inkling 2 Mini running on dual RTX 3090s using NInfer-all, without llama.cpp.
557K context at ~100 tok/s on two cards.
174K context at ~127 tok/s on one card.
RAM prefix caching is the real win. A new chat using the same 63K-token document started in just 0.85 seconds.
Hardware and software
2× RTX 3090 (no NVLink, no P2P)
128 GB RAM
NVIDIA driver 615.71 / CUDA 13.4
NInfer-all 2026-11-21.3
Power limits: 320 W on the KV card, 200 W on the other.
Converting GGUF to .ninfer took approximately 10 seconds on CPU. WaveCut also publishes ready-to-use .ninfer files on Hugging Face.
1. Context capacity
The KV cache for this model family is BF16 only (no INT8/FP8), consuming approximately 36 KiB per token.
All KV cache resides on the first --devices GPU. Only the routed experts are distributed across both cards.
The strategy is therefore to leave GPU 1 for KV cache and push as many experts as possible to GPU
2.Maximum context with vision + DSpark:
One RTX 3090: 174,080 tokens
Two GPUs, default split: 360,448 tokens
Two GPUs, --tensor-split 1,15: 524,288 tokens
Two GPUs, --tensor-split 1,127 + overlay + 1024 prefill: 557,056 tokens
The 1,127 split leaves one of the 128 experts per layer on the KV GPU.
overlay keeps the vision tower in pinned RAM and borrows VRAM when processing images. Vision still works.
Interestingly, 561,152 also fits, but the server drops the prefill chunk to 512, making prefill approximately 11% slower.
I chose to sacrifice 4K tokens for better performance.
The model’s native context window is 1M tokens, so VRAM is the limiting factor.
2. Performance
Decode speed
One GPU (174K context):
DSpark enabled: 127 tok/s
No drafter: 105 tok/s
Code generation: up to 144 tok/s
Two GPUs (557K context):
DSpark enabled: 98–104 tok/s
Prefill speed
One GPU:
4K: 2,371 tok/s
32K: 1,307 tok/s
63K: 873 tok/s
162K: 367 tok/s
Two GPUs:
32K: approximately 1,140 tok/s
63K: approximately 700 tok/s
Two GPUs are slower because every MoE block crosses PCIe through host RAM. There is no P2P communication.
Things I tested that didn’t help:
Larger prefill chunks on one GPU.
--prefill-cublas.
More DSpark drafts (5 adaptive: 93.6 tok/s; 4 fixed: 67.7 tok/s).
--lookup-ngram.
Minimum-p adjustments.
--tensor-parallel-prompt-partials and --tensor-attention positions are currently Qwen-specific.
The defaults are already the fastest.
3. Prefix caching: The real breakthrough
This is arguably the most interesting result.
I loaded a 63K-token document into a chat. Initial processing took approximately 90 seconds to first token.
Then I opened three unrelated chats.
Finally, I started a completely new chat with the same document.
Result: 0.85 seconds TTFT.
The server reused 62,973 of 62,996 tokens from the prefix cache.
Configuration:
--host-cache-mib 32768
For long-context workloads, this makes a substantial practical difference.
Correctness testing:
Needle-in-a-haystack: correct at 5K, 63K and 162K.
Multi-turn conversations over 63K tokens: correct.
4. How to run it
My working configuration:
ninfer-serve Inkling-2-Mini-UD-IQ4_XS-ninfer-v3.ninfer --model-id inkling-2-mini \
--vision --vision-residency overlay \
--devices 1,0 --tensor-parallel --tensor-split 1,127 \
--max-context 557056 --prefill-chunk 1024 --host-cache-mib 32768
5. Limitations and things not tested
This is a single-machine test with relatively short runs.
Still untested:
Needle retrieval near the full 557K context limit.
Two-GPU prefill beyond 63K.
Image processing when context is almost full.
Tool calls.
If anyone has managed to get Humming NVFP4 working on sm_86, I’d be interested to hear how.
Thanks to:
iamwavecut for NInfer-all.
Unsloth for the GGUFs.
Thinking Machines for releasing the open weights.
Would be interested to see what other people are achieving with dual 3090 setups.