veloxquant_mlx.cache.qfilters_cache module

Q-Filters-adapted KV cache — query-agnostic projection eviction.

Inspired by “Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression” (arXiv:2503.02812, preprint). Documented as “Q-Filters-adapted (VeloxQuant-MLX implementation)” — not a faithful port.

Scores each cached key by its projection onto a frozen per-head direction (the Q-Filter) and evicts the lowest-scoring tokens — attention-quality importance without computing attention and without a query at eviction time. The repo’s fourth eviction scorer class: not attention/proxy (SnapKV, H2O, TOVA, PyramidKV, SqueezeAttention, ChunkKV, CaM), not structural (StreamingLLM, sink), not intrinsic-norm (L2Norm).

TWO FILTER SOURCES

Pass filters ([H_kv, D], from veloxquant_mlx.quantizers.qfilters_calibration) to use the paper’s query-SVD direction — the mechanism §3.2 actually specifies. The filter is then frozen before the first token, so the sign is correct by construction (Theorem 3.3’s kappa^h > 0), eviction runs from token 0 with no calibration window, and the kept set is path-independent: prefill in one block and token-by-token decode return bit-identical results.

With filters=None the cache falls back to estimating the direction from the SVD of the first qfilters_calib_tokens observed KEYS. That recovers the dominant axis but not its orientation — the sign is exactly what a query disambiguates — so on this path qfilters_sign is a genuine ablation knob, and the kept set is path-DEPENDENT (the direction depends on whichever chunk first crosses the threshold). Nothing on the fallback path is claimed equivalent to the paper’s filter.

Limitations (stated plainly):

• The anisotropy/attention-prediction claim is the paper’s, about trained models; nothing here validates it on real weights.

• No RoPE position-ID renumbering after eviction. Surviving tokens keep their original absolute positions, so self.offset reports the true token position and RoPE stays correct without re-rotating survivors (see update_and_fetch and :issue:`171`). Positions do become non-contiguous where tokens were dropped.

• Uniform budget and n_sink across all heads.

• qfilters_recent (trailing protected window) is an extension, off by default. With it off, a long prefill’s trailing tokens are ranked like any other and mostly evicted before generation starts — pure projection ranking has no recency bias. Set it when generation quality on long prompts matters (see qfilters_update and :issue:`172`).

• Calibrated filters are model-specific; a head-layout mismatch raises.

Byte accounting (same names as L2NormKVCache):

qfilters_kept_bytes — fp16 bytes for retained K + V (plus the frozen

float32 filter direction, once it exists)

full_seq_bytes — hypothetical fp16 cost if all tokens were kept compression_ratio — full_seq_bytes / qfilters_kept_bytes (> 1 = savings) tokens_seen — total token positions ever passed to update_and_fetch tokens_kept — tokens currently in the (B=0, H=0) head’s cache

class veloxquant_mlx.cache.qfilters_cache.QFiltersKVCache(config, filters=None)

Bases: DeferredEvictionMixin, KVCache

KV cache implementing Q-Filters-adapted projection eviction for one layer.

Parameters:

• config (Any) –

  KVCacheConfig. Fields consumed: qfilters_budget (int, default 512) — max tokens kept (incl. sinks), qfilters_n_sink (int, default 4) — leading positions never evicted, qfilters_recent (int, default 0) — trailing protected window (extension), qfilters_calib_tokens (int, default 128) — fallback only: tokens

  observed before the filter direction is estimated and frozen,

  qfilters_sign (int, default 1) — +1 = paper direction; -1 = inverted. qfilters_min_retention (float | None, default 0.9) — warn (or raise,

  see qfilters_on_low_retention) the first time retained tokens fall below this fraction of tokens seen. A budget sweep against a real long-context prompt (docs-site blog: “qwen3-8b-qfilters-budget-sweep”) found generation collapses into fully repeated, non-language output anywhere from 23% to 80% retention, then recovers sharply between 80% and 92% — this is a warning about a measured failure mode, not a quality guarantee at or above the threshold, and not proven universal across prompts or models. None disables the check entirely.

  qfilters_on_low_retention ("warn" | "raise" | "ignore", default

  "warn") — what happens the first time the retention floor above is crossed. "raise" turns a silently-degraded generation into a hard failure at the point it starts happening, instead of a normal looking response that just happens to be unusable.

  use_metal_kernels (bool | None, default None) — three-state Metal

  fast-path flag for the calibrated/batched path (see Notes below): None auto-detects, True requires Metal (raises at construction if unavailable or if qfilters_budget exceeds QFILTERS_MAX_BUDGET), False forces the pure-MLX path.

• filters (mx.array | None) – Optional [H_kv, D] pre-calibrated query-SVD Q-Filters for this layer (see veloxquant_mlx.quantizers.qfilters_calibration). When given, eviction is active immediately and path-independent; when omitted, the key-SVD fallback applies. Must match the layer’s (H, D) — a mismatch raises on the first update_and_fetch rather than silently mis-evicting.

Notes

No .bits attribute — stores and returns fp16 K/V directly. Single-layer (no coordinator); the default KVCacheBuilder.for_model() path returns one QFiltersKVCache per attention layer. Per-head state is lazily initialised on the first update_and_fetch. Validation (sign, sink/recent-vs-budget) happens at construction. Writes through to the base mlx_lm KVCache’s self.keys / self.values / self.offset on every call so .state stays valid (mlx_lm’s generate() reads it unconditionally during chunked prefill); is_trimmable() reports False since the internal per-token state can’t be rolled back by a base-class trim() (see #83).

With calibrated filters, eviction runs as ONE vectorized selection over all B*H groups (qfilters_update_batched) instead of a Python loop per (batch, head) — the loop cost scaled with B*H and dominated decode at realistic head counts (:issue:`173`). Selection is bit-for-bit what the loop produced. Per-head QFiltersState objects are then stale, and are rebuilt lazily by _sync_states() for the diagnostic surface (states); nothing on the hot path reads them. The key-SVD fallback keeps the loop — its per-group filters freeze at different times, which a single shared-filter selection cannot express.

On top of that batched selection, when Metal is available (or use_metal_kernels=True) and the group is actually over budget with qfilters_budget <= QFILTERS_MAX_BUDGET, the over-budget branch dispatches to qfilters_fused_evict — two fused Metal kernels (scoring + threshold compaction, see veloxquant_mlx/metal/_qfilters_evict.py) that replace the pure-MLX mx.argsort / mx.take_along_axis selection with the same tie-breaking convention, so the two paths agree bit-for-bit (:issue:`179`). Under-budget calls and the key-SVD fallback loop are unaffected either way — there is nothing to evict yet, or the shared batched selection this optimizes does not apply to them.

update_and_fetch(keys, values)

Absorb new K/V tokens, apply projection eviction, return retained window.

Parameters:

• keys (array) – [B, H, S, D] new key tokens (any dtype; cast to fp16).

• values (array) – [B, H, S, D] new value tokens.

Returns:

Full prior kept rows plus all new rows, before eviction (#610). Only the state stored for the next call is compressed.

is_trimmable()

False: trim() would only roll back base-class offset bookkeeping, not the internal per-token eviction/compression state that actually determines what gets returned, silently corrupting future calls.

Return type:

bool

property qfilters_kept_bytes: int

Bytes currently stored across all heads (fp16 K + V + filter dirs).

property full_seq_bytes: int

Hypothetical fp16 K + V cost if all tokens were kept.

property compression_ratio: float

full_seq_bytes / qfilters_kept_bytes; > 1 means memory savings over fp16.

property tokens_seen: int

Total token positions ever passed to update_and_fetch (all heads summed).

property tokens_kept: int

Tokens currently in the (B=0, H=0) head’s cache (diagnostic).

property states: list[QFiltersState]

Per-head eviction states (diagnostic).

On the batched path these are rebuilt lazily from the [BH, n, D] buffers, so reading this is what pays the BH-length cost — never the hot path.

property merge