veloxquant_mlx.cache.anchorkv_cache module

AnchorKV-adapted KV cache — anchor-residual compression, no eviction.

Inspired by “AnchorKV: Anchor-Residual KV Cache Compression” (Khalaf, Shamshoum, Hodos, Sieradzki, Schuster; Technion; arXiv:2608.02901v1, 2026-08-03). Documented as “AnchorKV-adapted (VeloxQuant-MLX implementation)” — not a faithful port. No verified peer-reviewed venue as of 2026-08-20 — a one-time, user-directed exception to this repo’s venue-verification rule (the same exception previously granted to NestedKV). See paper/research/surveys/NEW_METHOD_SURVEY_V22.md and quantizers/anchorkv.py for the full list of adaptation decisions.

Unlike every eviction method in this repo (H2O, SnapKV, PyramidKV, AdaKV, NestedKV, …), AnchorKV never drops a token: every position stays inside the softmax. At the end of prefill, each head selects a small set of anchor positions (stored exactly), assigns every other token to its nearest anchor (one index + one scalar coefficient), and spends a byte budget derived from anchorkv_theta on quantized residuals for whichever tokens’ approximation error costs the most attention-output error (a first-order utility estimate, paper Eq. 6). Decode tokens are appended exactly at fp16, never retroactively anchored — the same one-shot-prefill convention as SnapKV-adapted / NestedKV-adapted.

This module reconstructs a DENSE fp16 K/V tensor on every call (prefill and decode alike) and hands it to the base mlx_lm KVCache — the paper’s fused tiled-reconstruction kernel (never materializing the dense cache) is NOT implemented here; this is a correctness-first adaptation, and the byte accounting below reflects the COMPRESSED representation’s true storage cost, not the transient dense tensor this wrapper materializes to satisfy the update_and_fetch contract.

Byte accounting:

anchorkv_bytes — true compressed storage: anchors (fp16) + per-token

metadata (index + coefficient) + packed residuals, summed over all heads

full_seq_bytes — hypothetical fp16 K + V cost for the same tokens compression_ratio — full_seq_bytes / anchorkv_bytes (> 1 = savings) tokens_kept — always == tokens_total (no eviction, ever)

class veloxquant_mlx.cache.anchorkv_cache.AnchorKVKVCache(config)

Bases: KVCache

KV cache implementing AnchorKV-adapted anchor-residual compression for one layer.

Parameters:

config (Any) –

KVCacheConfig. Fields consumed: anchorkv_theta (float, default 0.05) — fraction of the

uncompressed fp16 cache to retain; the single user-facing compression knob (paper §3.4).

anchorkv_window (int, default 32) — trailing positions

always kept as anchors and used as proxy observation queries.

anchorkv_rho (float, default 0.7) — fraction of the

non-window anchor budget filled by attention score (the rest sampled uniformly).

anchorkv_anchor_frac (float, default 1/128) — anchor budget

k as a fraction of context length S (paper: k = S/128).

anchorkv_residual_bits (int, default 2) — bits/coordinate

for stored residuals.

anchorkv_seed (int, default 42) — RNG seed for the

uniform anchor share and the residual codec’s rotation.

Notes

No .bits attribute — stores and returns fp16 K/V directly (the anchor-residual representation is internal; update_and_fetch always returns a dense reconstruction). Compression happens ONCE at prefill (S > 1), across all heads independently for anchor selection/assignment but with the residual budget allocated by pooling utilities ACROSS heads (paper §3.4). Decode tokens (S == 1) are always appended exactly — never dropped, never retroactively anchored. Single-layer (no coordinator); KVCacheBuilder.for_model() propagates all anchorkv_* fields automatically via dataclasses.replace.

update_and_fetch(keys, values)

Absorb new K/V tokens; prefill compresses once, decode always appends exactly.

Parameters:

• keys (Any) – [B, H, S, D] new key tokens (any dtype; cast to fp16/fp32 internally).

• values (Any) – [B, H, S, D] new value tokens.

Returns:

(K_out, V_out) both [B, H, n_total, D] fp16 — n_total is always the full token count seen so far, never reduced.

is_trimmable()

False: trim() would only roll back base-class offset bookkeeping, not the internal anchor/residual state that determines what gets returned, silently corrupting future calls.

Return type:

bool

property anchorkv_bytes: int

anchors + metadata + packed residuals, all heads.

Type:

True compressed storage

property full_seq_bytes: int

Hypothetical fp16 K + V cost for the same tokens, uncompressed.

property compression_ratio: float

full_seq_bytes / anchorkv_bytes; > 1 means storage savings.

property tokens_kept: int

Always equals tokens_total — AnchorKV never evicts a token.

property tokens_total: int

Total token positions ever passed to update_and_fetch (all heads summed).

property n_anchor_total: int

Total anchor positions selected across all (batch, head) pairs at prefill.

property n_residual_total: int

Total residual slots spent (K + V) across all (batch, head) pairs at prefill.