Project · Local AI

Localweights

Quantised Qwen models in GGUF format for local inference, published by Nick Stocks on Hugging Face.

What I publish

The Crucible collection includes GGUF variants of Qwen models in sizes from 4B to 35B-A3B. I publish variants at different quantisation levels so you can choose a build around your available memory and expected context length.

Example release: Qwen3.6 27B MTP GGUF
Weight quantisation: IQ4_XS with an importance matrix
MTP tensors: Q8_0 override for the NextN tensors in this variant

Multi-token prediction (MTP) can propose draft tokens for the main model to verify. Runtime support is model-specific. Check the model card for the required llama.cpp build and settings.

View the Crucible release collection

Decode on an RTX 3090 Ti

Qwen3.6 27B release benchmark at a 192K configured context, with Q8 key and value cache. Mean of five warm runs, 512 generated tokens per run.
Configuration	Tokens/s
Greedy decode	~48
MTP speculative decode	~73

These are single-stream decode measurements with full GPU offload. See the benchmark settings in the model card.

How I compare builds

Crucible is my local inference testbed.

Read the local Qwen benchmarking note for a repeatable comparison of memory use, prompt processing and decode speed.