Conversation about Quantization(Also about hosting DeepSeekv4:Flash)

A 284-billion-parameter AI model should not fit on one local machine.

DeepSeek V4 Flash does.

In Episode 20 of System Prompt, Peter and Val explore how quantization, mixed precision, importance-aware compression, and speculative decoding make it possible to run a massive mixture-of-experts model on hardware such as a single DGX Spark.

Peter breaks down how Antirez compressed DeepSeek V4 Flash to about 81 GB while preserving enough reasoning, coding, and tool-use ability to remain useful.

Most parameters sit inside routed experts using roughly two-bit quantization. More sensitive components remain at Q8, FP16, or FP32. An importance matrix helps identify which compression errors are most likely to damage the model's behavior.

Peter also demonstrates the model running live through his local agent infrastructure at roughly 20 to 30 tokens per second.

WHAT WE DISCUSS

• Running DeepSee