Prefill and decode do not have to run the same weights. This repo holds prefillers: NVFP4 transformer blocks that are streamed off the SSD one layer at a time, paired to complement low-bit GGUF for decode.
The point is that the two phases want different things. Decode is memory-bound, has to be on device for fast generation and is accelerated by being small. Prefill is compute-bound, so there’s little point in compressing it below NVFP4 (compute-native on Blackwell) as long as it doesn’t occupy much memory, which we achieve via offloading.
Accuracy and time-to-first-token against device weight size
Auxiliary prefillers recover ~50% of the accuracy that aggressive quantization gives up, at close to no extra memory cost. Moreover, past a few thousand tokens, the offloading streaming disappears under compute, so it is not paid for in latency either
That’s actually a great idea.
It’ll need a big batch size to hide the latency though (so this only worth it past 8K context, or so?) and then it needs to reload the quantized weights onto the GPU.
It does suck that all paths to better inference are so bad. Bigger GPUs are crazy expensive. Good IGPs are rare. CPU RAM is cost prohibitive. Even NVMe drives are crazy expensive now. Hence there’s only so much that can be done if users have absolutely no options to buy.
But there’s good news, the whole scientific field seems to put a lot of effort into better quantization, which means you can run better models with lower resources. The last big innovations were:
- MoE
- KV Chache quant
- turboquant (rarely implemented)
- ternary quant (prism-ml)
So I am just waiting for stream smaller weights for the same quality, giving me better results in the end.
Ternary quant from prism-ml is not that good for complex task and long context. Probably fixable by using higher bit on certain tensor. As far I know for real world usage, at least non uniform 3 bit base is needed.
If you are on Nvidia, have you considered using exllama/TabbyAPI with an MoE model?
It already does something more practical: splitting the MoEs in each layer, profiling them, and keeping the “hot” ones on the GPU. Its quantization is also state of the art, way better than llama.cpp, and it’s had KV hadamard rotation implemented years before the TurboQuant.
Interesting, will check that out too.
Just for reference, I’m now getting 20 tokens/sec with Deepseek V4 Flash 0731, on a single 3090 desktop, whereas I was only getting 13 in ik llama.cpp, and less in base llama.cpp.
I have not switched to 4.1 in IK yet. All these releases are happening so fast by brain can’t even keep up…
If that’s true that’s insane on many levels
I’m on amdgpu personally and found beellama to be a nice step in this direction FWIW
Be careful with beellama, as it’s largely vibecoded from what I understand.
That’s not bad per se, but you have to watch out for implementation quirks (eg a model might not inference quite correctly) or questionable features (like TurboQuant, which is sorta obsolete with newer models, and seems to benchmark questionably on older ones).
There are some interesting AMD forks out there though. I wish I could speak to some, but I haven’t tested any, and it’s hard to keep up.
