Colibrì: Running GLM-5.2 (744B) Locally in RAM With No GPU

Summary

Key Points

• —The video showcases running GLM 5.2, a 744 billion parameter Mixture of Experts (MoE) model, on a single machine without a cluster or multiple GPUs. 0:28
• —This feat is accomplished using Colibri, a tiny C engine with zero dependencies that streams model components from disk on demand. 0:41
• —The core principle leverages the fact that the model needs about 370 GB on disk but only a fraction of that (around 10 GB for dense layers plus cached experts) in RAM at any given point. 1:27
• —As an MoE model, GLM 5.2 only activates a small subset of its 21,050 experts (e.g., 8 out of 256 per layer) for each token, rather than using all parameters simultaneously. 2:29
• —The system's speed improves with continued use because Colibri intelligently caches frequently hit experts in RAM, leading to a higher 'expert hit rate' and faster token generation. 3:01
• —The primary performance bottleneck for this system is the disk read speed, not the CPU or GPU, as the GPU remains largely idle during the generation process. 3:23
• —Colibri manages memory by keeping the dense model parts and frequently accessed 'hot' experts in RAM, while streaming other necessary experts directly from disk. 4:06
• —This streaming technique is specifically effective for Mixture of Experts models because only a small portion of the model is required per token, making it impractical for dense models. 9:30
• —While the current Colibri implementation is handwritten and optimized for GLM 5.2's exact architecture, the underlying technique is general and holds promise for other MoE models. 9:47