AirLLM: Running 70B Models on 4GB of VRAM Changes the Local AI Game
AirLLM is a project that makes it possible to run 70-billion-parameter language models on consumer hardware — specifically, on a single GPU with as little as 4GB of VRAM. The technique is layer-by-layer inference, which loads and processes one model layer at a time rather than the entire model at once. The practical implication is … Read more