I posted earlier about getting Inkling-2-Mini running on my own inference engine the day after it came out. This is the follow up with a new image and a lot more numbers. Short version: it now runs at a 524,288 token context on one 24 GB Radeon, and the 262k numbers went up quite a bit.
Hardware for everything below: RX 7900 XTX 24 GB, i7-9700, 32 GB RAM, Linux, Vulkan. One card, whole model on the card, nothing in system RAM. 4-bit KV cache on both engines. Single runs, greedy decoding, nothing else running on the box.
The model file is 17.6 GB. It keeps the experts exactly as Thinking Machines shipped them (their own 4-bit format, bit identical to the checkpoint) and the rest is 8-bit. Both engines ran that same file.
BENCHMARKS, LexiEngine vs llama.cpp b12600, same card, same file
4,096 token prompt
LexiEngine: 2,245 tok/s prompt, 81.7 tok/s generation with the draft head (52.4 without it)
llama.cpp: 425 tok/s prompt, 25.4 tok/s generation
16,384 token prompt
LexiEngine: 2,148 tok/s prompt, 80.4 tok/s generation with the draft head (50.9 without it)
llama.cpp: 397 tok/s prompt, 25.0 tok/s generation
261,632 token prompt (262k context)
LexiEngine: 1,146 tok/s prompt (3 min 48 s), 40.2 tok/s generation
llama.cpp: did not finish the prompt in 55 minutes
523,264 token prompt (524k context)
LexiEngine: 767 tok/s prompt (11 min 23 s), 33.7 tok/s generation
llama.cpp: not run
MEMORY AT 524k
weights 17.62 GB
KV cache 5.02 GB
scratch 0.58 GB
total in use on the card: 23.7 of 25.75 GB
Load time is 13 seconds. A 1M context does not fit on 24 GB with this layout (the cache alone would be 9.85 GB), so 524k is the ceiling on this card for now.
DOES IT ACTUALLY READ THE CONTEXT
I planted ten facts through the prompt at 1%, 10%, 20% and so on up to 99%, then asked for each one back.
at 260,864 tokens: 10 of 10 correct
at 523,264 tokens: 10 of 10 correct
That is the test I trust more than a perplexity number at these lengths, because my CPU reference cannot run half a million tokens in any reasonable time.
DRAFT HEAD (the model ships its own)
Output is identical to plain greedy decoding in every run I checked, it only changes speed.
after 4k tokens: 81.7 tok/s (about 6 tokens per pass of the main model)
after 16k tokens: 80.4 tok/s
at 262k: 41.8 tok/s vs 40.2 plain
at 524k: 32.1 tok/s vs 33.7 plain, so it is slower there
served at 524k context with a short prompt: 51.0 tok/s vs 43.9
So it is a big win at short and medium depth and a small loss once you are half a million tokens deep, because the draft head's own attention over that much context costs more than it saves. I left that number in on purpose. It only applies to greedy requests in this build, and you have to point the instance at the model's checkpoint folder to turn it on (the draft head is a separate file from the GGUF).
WHAT CHANGED SINCE THE LAST IMAGE (262k context)
prompt: 791 -> 1,146 tok/s (+45%)
generation: 33.3 -> 40.2 tok/s (+21%)
Most of the prompt gain came from the attention kernel for the four layers that see the whole context. That was about half of the prompt time and I had not touched it before.
ACCURACY
Against a CPU reference of the published weights, 510 positions, 4-bit KV:
LexiEngine prompt path: mean KL 0.0038
LexiEngine token by token: mean KL 0.0043, same top token 98.6%
llama.cpp: mean KL 0.0037, same top token 98.8%
So accuracy is a tie, the difference is speed. With the KV cache left unrounded my engine matches the reference exactly (KL 0.00000 over 1,024 positions), which tells me the remaining error is the 4-bit cache and not a bug. For reference the 4-bit cache costs about 0.005 mean KL against about 0.002 for 8-bit on this model.
THINGS YOU SHOULD KNOW BEFORE TRUSTING ANY OF THIS
The llama.cpp numbers were measured the day before, with a disk copy running at the same time, and I have not re-run them. They could be a bit low. I do not think it changes the picture at 4x to 5x but I would rather say it.
llama.cpp's stock converter decodes the 4-bit experts and re-quantizes them, which gives a 27.8 GB file at 8 bits and does not fit a 24 GB card. To compare on the same card I used a file that keeps the experts as shipped, and llama.cpp needed a small loader patch to run that file correctly. The patch is printed in full on the wiki page linked below.
Everything is single runs on one machine. If you have a 7900 XTX I would genuinely like to know if you get the same numbers.
Not in this build: image input, the draft head for sampled (non greedy) requests, and a 1M context.
WHAT THIS IS
LexiSuite is my own stack, written in Rust: inference engine, control panel, a bootable OS image. It is not a wrapper around llama.cpp or Ollama. This is a development image, not a production release. The USB image has no models in it and writing it erases the stick.
Also new in this build if you care: prompt reuse between requests (a follow up turn only processes the new tokens), reasoning separated from the answer in the chat API, and LoRA adapter loading for this model.
Download and USB instructions: https://lexipanel.com/download/
Benchmarks and method: https://github.com/W61k3r/LexiSuite-Releases/wiki/Benchmarks
How the Inkling runtime works, with the llama.cpp patch: https://github.com/W61k3r/LexiSuite-Releases/wiki/LexiEngine-Inkling-2
Releases: https://github.com/W61k3r/LexiSuite-Releases/releases