Technical note · Local inference
Benchmarking Qwen GGUF models locally
A practical way to compare quantisation, context memory and speculative decoding on the same hardware.
Compare like for like
Start with two quantisations of the same base model. Hold the runtime build, GPU offload, prompt, sampling settings and output length fixed. Fit the test to your intended workload with a short interactive prompt and a second prompt close to your expected context size.
Measure the parts separately
- Record cold load time, then measure warm time to first token with the model already loaded.
- Record prompt processing and generation throughput separately. Repeat with a cold prompt cache and a reused prefix if that matches how you work.
- Record peak VRAM and system RAM. A build that fits at a short context may spill model layers or cache to host memory once the prompt grows, depending on runtime settings.
- Run a small task suite and check answers for regressions before choosing the smallest build.
For a build with MTP, repeat the same test with speculative decoding off and on. Track accepted draft tokens alongside throughput. Acceptance and speed can vary with workload, so measure both on prompts you actually use.