How Much RAM for Local LLMs?

1. The Hardware Bottleneck: VRAM vs. System RAM

When self-hosting open-weight models (like Llama 3 or Mistral), output speed is dictated by **memory bandwidth**. During inference, the GPU must fetch billions of parameters from memory to process each word. Dedicated graphics memory (VRAM) operates at massive bandwidths: an Nvidia RTX 4090 moves data at 1,008 GB/sec. Standard system RAM, however, operates at only 50 to 90 GB/sec.

If a model's size fits entirely inside VRAM, generation is extremely fast (30 to 80 tokens per second). If the model exceeds the VRAM ceiling and spills into system RAM, the GPU must fetch weights over the slow PCIe system bus, causing output speed to crawl to 1-3 tokens per second—making it unusable for real-time applications.

2. The Math of Model Compression: Quantization

By default, models are trained at 16-bit precision (FP16), consuming 2 bytes of memory per parameter. A 7B parameter model at FP16 requires 14 GB of memory just to load the weights. A 70B parameter model requires 140 GB, putting it far out of reach for consumer GPUs.

Quantization solves this by converting 16-bit float values into lower-bit representations (like 4-bit or 5-bit integers) using mapping algorithms. This reduces memory footprint dramatically:

• 16-bit (FP16): 2.0 bytes per parameter. (7B model = 14.0 GB weights)
• 8-bit (Q8_0): 1.0 byte per parameter. (7B model = 7.0 GB weights)
• 4-bit (Q4_K_M): ~0.5 bytes per parameter. (7B model = 3.5 GB weights)
• 2-bit (Q2_K): ~0.25 bytes per parameter. (7B model = 1.75 GB weights)

Quantization reduces file sizes on disk and in memory, but introduces a slight math rounding error (measured as "perplexity loss"). Empirically, 4-bit quantization provides the ultimate sweet spot—reducing file size by 75% with a perplexity increase that is virtually imperceptible in conversation.

3. The KV Cache Memory Formula

In addition to model weights, runners must store Key-Value (KV) tensors for every token processed in the context window. This memory buffer is called the KV Cache. It prevents the model from re-evaluating conversation history from scratch on every turn.

The memory size of the KV cache scales linearly with context length and batch size:

For modern Grouped-Query Attention (GQA) models, a quick empirical estimate is:
KV_VRAM ≈ Context_Length × Batch_Size × Model_Size × 0.00000015 GB.

At 8,192 context length and batch size 1, a 70B model's KV Cache consumes ~1.0 GB of memory. However, if running a multi-user server at batch size 16 with 32,000 context limits, the KV Cache alone consumes over 16 GB of VRAM, requiring developers to size their graphics hardware accordingly.

4. Recommended Hardware Configuration Tiers

Sizing your hardware configuration depends on which parameter scales you plan to run:

Frequently Asked Questions