Running a 1T-Parameter
Model on a Laptop

Kimi-K2 has 1.04 trillion parameters. Stored in 4-bit precision, that footprint is 578 GB. The laptop has 128 GB of unified memory. The math is simple and brutal: the model is 4.5 times bigger than the machine.

Two separate problems need solving. First, those 578 GB of weights have to get into the compute path somehow. Second, as the model generates text, its attention memory (the "KV cache") grows with every token and eventually eats what little RAM you have. Two techniques handle this: SSD streaming for the weights, and TurboQuant for the cache.

Not all trillion parameters fire for every token. Kimi-K2 is a Mixture-of-Experts model. Think of a hospital with 384 specialist doctors on staff. A patient walks in, and a triage nurse (the "router") picks the 8 best specialists. Those 8 examine the patient, combine their opinions, and produce a diagnosis. The other 376 doctors stay in their offices doing nothing. That is 97.9% idle.

If almost everything is idle almost all the time, why keep it in memory?

The trick: leave expert weights on the SSD. When the router picks 8 experts, read just those 8 from disk using pread() -- a system call that reads bytes from a specific position in a file without moving a cursor. Each expert's weights sit at a known byte offset inside a safetensors file. Jump straight there, read the bytes, do the math, throw them away. Repeat for the next token.

def load_experts(self, expert_ids: list) -> Tuple[mx.array, mx.array, Optional[mx.array]]:
    """Load selected experts' weights from SSD.

    Uses C extension for parallel pread when available, falls back to Python.
    OS page cache handles caching -- 'Trust the OS' (Flash-MoE finding).
    Returns (weight, scales, biases) as mx.arrays with shape [K, ...].
    """
    try:
        return self._load_experts_fast(expert_ids)
    except (ImportError, Exception):
        return self._load_experts_python(expert_ids)

At 0.19 tok/s on Kimi-K2, the model ran but barely. Profiling pointed the finger at Python itself, not the SSD. Each token needs 60 layers times 3 projections = 180 pread() calls. Each call in Python creates a bytes object, converts to NumPy, converts to MLX -- three allocations, two copies. At ~5 ms overhead per call, that is 900 ms of pure interpreter tax per token.

The SSD can deliver 18 GB/s. It is not the bottleneck. Python is.

/* Thread argument for parallel pread */
typedef struct {
    int fd;
    void *buf;
    size_t size;
    off_t offset;
    ssize_t result;
} PreadTask;

static void *pread_thread(void *arg) {
    PreadTask *task = (PreadTask *)arg;
    task->result = pread(task->fd, task->buf, task->size, task->offset);
    return NULL;
}

/* Allocate output buffer for ALL experts at once */
size_t total_bytes = (size_t)k * expert_bytes;
void *buf = malloc(total_bytes);

/* Launch parallel pread threads (up to 4) */
for (Py_ssize_t i = 0; i < k; i++) {
    tasks[i].fd = fd;
    tasks[i].buf = (char *)buf + i * expert_bytes;
    tasks[i].size = expert_bytes;
    tasks[i].offset = base_offset + (off_t)expert_ids[i] * expert_bytes;
}

/* Create numpy array that OWNS the buffer -- zero copy */
PyObject *arr = PyArray_SimpleNewFromData(1 + shape_len, dims, dtype_num, buf);
PyArray_ENABLEFLAGS((PyArrayObject *)arr, NPY_ARRAY_OWNDATA);

Three wins over pure Python: single allocation (one malloc() for all experts), parallel pread with pthreads (up to 4 threads hitting the NVMe simultaneously), and zero-copy return (the buffer becomes the NumPy array directly via NPY_ARRAY_OWNDATA).

With correctness confirmed on the small model, streaming scales to the ones that actually need it.

When a language model generates text, it uses attention to figure out which earlier tokens matter for the next one. Every processed token produces three vectors:

• Query (Q) -- "What am I looking for?"
• Key (K) -- "What do I contain?"
• Value (V) -- "Here is my content."

The current token's Query is compared against every previous Key to find matches. The matching Values are combined into context. This is how the model refers back to earlier parts of the conversation.

The problem: to generate token #1000, you need all 999 previous Keys and Values in memory. At token #128,000, you need 127,999 of them. This stored history -- the KV cache -- grows linearly. For Kimi-K2 at 128K tokens, it can reach 4-8 GB in 16-bit precision.

That 4-8 GB is RAM that could otherwise cache frequently-used expert weights, making SSD streaming faster. Compressing the KV cache helps both memory and speed.

Quantization means reducing numerical precision to save space. A 16-bit number can represent 65,536 distinct values; a 4-bit number gets 16. If you can round to fewer bits without losing important information, memory shrinks proportionally: 16-bit to 3-bit = 5.3x smaller.

Standard quantization divides the value range into equal-sized bins. But real KV cache data has outliers -- occasional extreme values far larger than typical ones. These outliers stretch the bins across a huge range, wasting resolution on values that rarely appear.

The core insight from Google Research's TurboQuant (paper): rotate the data first and the outliers disappear.

A Hadamard matrix is a special square matrix of +1s and -1s (scaled by a normalization factor). Multiplying your data by it is a rotation that redistributes energy from a few extreme dimensions across all dimensions equally. The operation is lossless -- it is an orthogonal transformation, meaning you can always rotate back to recover the exact original.

def create_hadamard_matrix(dim: int) -> mx.array:
    """Create a normalized Hadamard-like rotation matrix.

    Uses the recursive Walsh-Hadamard construction for powers of 2.
    The rotation transforms arbitrary distributions into near-Gaussian,
    which is the key insight of TurboQuant.
    """
    if dim == 1:
        return mx.array([[1.0]])

    # Walsh-Hadamard: double the matrix recursively
    #   [H  H]
    #   [H -H]
    H = mx.array([[1.0]])
    while H.shape[0] < dim:
        H = mx.concatenate([
            mx.concatenate([H, H], axis=1),
            mx.concatenate([H, -H], axis=1),
        ], axis=0)
    return H / math.sqrt(dim)  # Normalize to make orthogonal

After rotation, kurtosis drops from 27.7 to 3.2 -- nearly identical to a perfect Gaussian (3.0). The outliers are gone, and standard quantization works beautifully.

The rotation matrix is lossless but quantization is not. The question that matters: after replacing the standard KV cache with a rotate-quantize-dequantize-derotate pipeline across all 10 full-attention layers, does the model still produce correct answers?

def update_and_fetch(self, keys, values):
    """Store new keys/values (compressed) and return all cached K,V (decompressed)."""
    B, n_kv_heads, num_steps, k_head_dim = keys.shape

    # Create rotation matrices on first use
    if self._k_rotation is None:
        self._k_rotation = create_hadamard_matrix(k_head_dim)
        self._v_rotation = create_hadamard_matrix(v_head_dim)

    # ROTATE then QUANTIZE (the TurboQuant innovation)
    keys_q = mx.quantize(
        keys @ self._k_rotation,
        group_size=self.group_size, bits=self._tq_bits)
    values_q = mx.quantize(
        values @ self._v_rotation,
        group_size=self.group_size, bits=self._tq_bits)

    # ... store quantized data ...

    # DEQUANTIZE then DE-ROTATE for return
    k_deq = mx.dequantize(*self.keys, group_size=self.group_size, bits=self._tq_bits)
    v_deq = mx.dequantize(*self.values, group_size=self.group_size, bits=self._tq_bits)
    k_deq = k_deq @ self._k_rotation.T   # De-rotate (R^T = R^-1 for orthogonal R)
    v_deq = v_deq @ self._v_rotation.T

    return k_deq, v_deq

The flow: rotate → quantize → store → dequantize → de-rotate → return. The only error source is quantization, and it is operating on near-Gaussian data now.

References

• TurboQuant Blog -- Google Research
  Overview of rotation-based KV cache compression with zero quality loss.
• TurboQuant Paper (arXiv:2504.19874)
  Full paper: rotation + quantization + QJL correction for extreme compression.
• PolarQuant (arXiv:2502.02617)
  Related work: KV cache quantization using random rotation matrices.
• Flash-MoE (GitHub)
  SSD-based MoE inference. Key finding: "Trust the OS page cache."
• MLX (GitHub)
  Apple's machine learning framework for Apple Silicon.