veloxquant_mlx.cache.sliding_window_cache module

Sliding-window eviction decorator wrapping any standalone KVCache.

SlidingWindowKVCache composes over another KVCache instance to cap it at the most recent window_size tokens. Because the wrapped caches (TurboQuant, PolarQuant, QJL, etc.) don’t support random deletion, eviction is implemented by keeping raw key/value vectors in ring buffers and, each time the window advances, resetting the inner cache to empty (via its own reset()) and re-feeding it the current window — correct but O(window) per eviction, so it targets inference rather than training. Only compatible with STANDALONE_METHODS (see KVCacheFactory.create’s sliding_window handling).

class veloxquant_mlx.cache.sliding_window_cache.SlidingWindowKVCache(inner, window_size)

Bases: KVCache

Wraps any KVCache with sliding-window token eviction.

Maintains a RingBuffer of the most recent window_size token slots. When the window is full, the oldest token is evicted.

Because the inner cache does not support random deletion, we maintain the window by resetting the inner cache to empty and re-feeding it the current window’s tokens each time one is evicted. This simple strategy is correct but rebuilds the cache every window_size tokens; suitable for inference not training.

We cache all key-value vectors in our own ring buffer so they’re available to re-feed on each window advance.

Parameters:

• inner (KVCache) – The underlying KVCache to wrap.

• window_size (int) – Number of tokens to keep.

append_key(k)

Buffer a key vector for paired insertion with the next value.

Parameters:

k (Any) – Key vector, shape (d,), fp16.

Return type:

None

append_value(v)

Insert a key-value pair and evict the oldest token if full.

Parameters:

v (Any) – Value vector, shape (d,), fp16.

Return type:

None

attend(q)

Delegate to the inner cache for attention computation.

Parameters:

q (Any) – Query vector, shape (d,), fp16.

Return type:

Any

Returns:

Attention output, shape (d,), fp16.

memory_bytes()

Return memory usage of the inner (windowed) cache.

Return type:

int

reset()

Clear the window and the inner cache, returning both to empty.

Return type:

None