veloxquant_mlx.cache.base module

Central KV-cache configuration, factory, and per-model builder.

Defines KVCacheConfig (one dataclass holding every method’s hyperparameters — over 40 quantization/eviction methods share this single config surface, each namespacing its own fields with a method-specific prefix), KVCacheFactory (dispatches config.method to the matching concrete cache class), and KVCacheBuilder (a fluent builder plus for_model(), which constructs one cache per language-model layer — handling per-layer bit-width schedules, hybrid-attention models, and methods that need a shared cross-layer coordinator such as XQuant, MiniCache, xKV, PyramidKV, CacheGen, SqueezeAttention, and ChunkKV layer-reuse). STANDALONE_METHODS marks the methods whose cache implements VeloxQuant’s own KVCache ABC instead of mlx_lm’s serving protocol, so for_model() can reject them with a clear error rather than failing deep inside generation.

class veloxquant_mlx.cache.base.KVCacheConfig(method='turboquant_rvq', head_dim=128, bit_width_inlier=2, bit_width_outlier=None, jl_dim=None, n_outlier_channels=None, n_calib_tokens=None, enable_vectorized_attend=True, enable_outlier_two_stream=False, enable_fused_query_dot=False, seed=42, dtype=None, capacity=None, sliding_window=None, store=None, observers=<factory>, options=None, key_sub_dim=4, value_sub_dim=8, key_codebook_bits=12, value_codebook_bits=8, residual_length=128, kivi_group_size=32, svdq_rank=None, svdq_energy_threshold=0.95, svdq_bit_schedule=(8, 4, 2, 1, 1, 0, 0, 0), svdq_group_size=32, kitty_hi_fraction=0.25, kitty_hi_bit=4, kitty_lo_bit=2, kitty_group_size=32, adakv_target_avg_bits=2.5, adakv_lo_bit=2, adakv_mid_bit=3, adakv_hi_bit=4, adakv_group_size=32, adakv_update_interval=1, adakv_importance='norm_variance', adakv_obs_window=32, xquant_group_size=2, xquant_base_bits=2, xquant_residual_bits=4, xquant_group_quant_size=32, xquant_max_ctx=8192, kvquant_bits=3, kvquant_outlier_fraction=0.01, kvquant_group_size=32, kvquant_lloyd_iters=8, kvquant_refit_interval=0, kvquant_n_sink=1, palu_rank=None, palu_energy_threshold=0.9, palu_n_head_groups=4, palu_hi_bit=4, palu_lo_bit=2, palu_hi_fraction=0.25, palu_group_size=32, palu_quantize_values=True, cachegen_bits=4, cachegen_group_size=32, cachegen_use_delta=True, cachegen_per_channel=True, cachegen_layer_groups=3, cachegen_resolved_bits=None, minicache_start_frac=0.5, minicache_group_size=2, minicache_retention_threshold=0.9, minicache_slerp_t=0.5, minicache_max_ctx=8192, gear_bits=2, gear_rank=None, gear_energy_threshold=0.9, gear_sparse_fraction=0.01, gear_group_size=32, gear_quantize_values=True, zipcache_hi_bits=4, zipcache_lo_bits=2, zipcache_hi_fraction=0.2, zipcache_group_size=32, zipcache_quantize_values=True, snap_dtype='auto', snap_batched_scoring=True, snap_backend='auto', snap_budget=512, snap_obs_window=32, snap_n_sink=4, stream_n_sink=4, stream_window_size=512, h2o_budget=512, h2o_n_sink=4, h2o_rope_base=10000.0, h2o_grace=16, h2o_decay=0.98, tova_budget=512, tova_n_sink=4, tova_backend='auto', pyramid_budget=512, pyramid_backend='reference', pyramid_n_sink=4, pyramid_beta=2.0, pyramid_resolved_budget=None, squeeze_budget=512, squeeze_n_sink=4, squeeze_strength=1.0, squeeze_resolved_budget=None, chunkkv_budget=512, chunkkv_chunk_size=8, chunkkv_n_sink=4, chunkkv_score='attn_mass', chunkkv_reuse_layers=1, cam_budget=512, cam_n_sink=4, cam_merge='sim_weighted', cam_merge_keys=False, cam_merge_gate=True, xkv_group_size=2, xkv_rank=None, xkv_energy_threshold=0.95, xkv_latent_bits=4, xkv_group_quant_size=32, xkv_max_ctx=8192, nsn_bits=2, nsn_residual_length=64, nsn_codebook_size=256, nsn_subvector_dim=8, nsn_seed=1234, nsn_max_ctx=8192, knorm_budget=512, knorm_n_sink=4, knorm_recent=0, knorm_keep='low', skvq_bits_key=2, skvq_bits_value=2, skvq_group_size=32, skvq_window=128, skvq_n_sink=5, skvq_reorder=True, skvq_clip_search=True, skvq_clip_alpha=1.0, skvq_max_ctx=None, qfilters_budget=512, qfilters_n_sink=4, qfilters_recent=0, qfilters_calib_tokens=128, qfilters_sign=1, qfilters_min_retention=0.9, qfilters_on_low_retention='warn', keyformer_budget=512, keyformer_n_sink=4, keyformer_recent=0, keyformer_tau=None, keyformer_tau_init=1.0, keyformer_tau_end=1.0, keyformer_anneal_steps=0, keyformer_rope_base=10000.0, keyformer_seed=0, morphkv_budget=512, morphkv_n_sink=4, morphkv_window=8, kvzip_budget=512, kvzip_n_sink=4, kvzip_probe='context', kvtc_bit_budget=512, kvtc_bit_choices=(0, 1, 2, 3, 4, 6, 8), kvtc_beta=3.5, curdkv_budget=512, curdkv_n_sink=4, curdkv_rank_cap=16, curdkv_rope_base=10000.0, nestedkv_budget=512, nestedkv_n_sink=4, nestedkv_window=64, nestedkv_beta=3.0, nestedkv_tau=0.6, nestedkv_kappa=10.0, nestedkv_safeguard_alpha=0.2, amc_k_high=0.2, amc_k_mid=0.3, amc_use_query_saliency=False, amc_query_alpha=0.5, amc_adaptive_thresholds=False, amc_threshold_window=64, amc_gamma=0.1, amc_calib_variance=None, amc_group_size=32, a2ats_codebook_bits=8, a2ats_sub_dim=8, a2ats_window=128, a2ats_b=2048, a2ats_use_query_aware=True, a2ats_beta=0.5, a2ats_retrieval_fraction=0.2, a2ats_rope_base=10000.0, a2ats_rope_scaling=None, a2ats_query_h=None, a2ats_codebook=None, anchorkv_theta=0.05, anchorkv_window=32, anchorkv_rho=0.7, anchorkv_anchor_frac=0.0078125, anchorkv_residual_bits=2, anchorkv_seed=42, rocketkv_compression_ratio=8.0, rocketkv_page_size=None, rocketkv_head_topk1=None, rocketkv_obs_window=32, rocketkv_n_sink=4, age_recent_boundary=128, age_mid_boundary=1024, age_bits_recent=8, age_bits_mid=4, age_bits_old=2, age_group_size=32, n_sink_tokens=5, smooth_factors=None, key_codebook=None, value_codebook=None, spectral_key_d_eff=4, spectral_val_d_eff=50, spectral_apply_qjl=True, spectral_model_name='model', use_metal_kernels=None, fused_sdpa=False, fused_sdpa_max_ctx=None, fused_sdpa_memory_bound=False)

Bases: object

Configuration for a KVCache instance.

Variables:

• method – Quantisation algorithm.

• head_dim – Attention head dimension (d).

• bit_width_inlier – Bit-width for inlier channels. Either a single int applied uniformly across all layers, OR a list[int] of length n_layers for per-layer RateQuant-style allocation. When passed as a list, KVCacheBuilder.for_model() consumes element i for layer i; KVCacheFactory.create() requires an int.

• bit_width_outlier – Bit-width for outlier channels (None → same as inlier).

• jl_dim – JL projection dimension m. None means m = head_dim for QJL, which is cheap but noisy (score correlation ~0.64 at d=128 vs ~0.88 at m=4d); raise it when attention quality matters. See #774.

• n_outlier_channels – Number of outlier channels to detect.

• seed – Random seed.

• dtype – MLX dtype for computations.

• capacity – Maximum tokens to store (None → unlimited).

• sliding_window – If set, wrap cache with sliding-window eviction.

• store – ArtifactStore to load precomputed artifacts from.

• observers – List of QuantizationObserver instances.

• options – Method-specific option dataclass from veloxquant_mlx.cache.options (e.g. H2OOptions, SnapKVOptions), grouping the flat per-method fields below (h2o_budget, snap_budget, …) into one small object per method. Preferred over setting the flat fields directly (#420): values from options populate the matching flat fields at construction time (see __post_init__), so every existing cache class – which reads the flat fields via getattr(config, "h2o_budget", 512) – keeps working unchanged. Setting a flat per-method field directly still works but emits a DeprecationWarning naming its options replacement.

Parameters:

• method (Literal['turboquant_prod', 'turboquant_mse', 'turboquant_rvq', 'polar', 'qjl', 'vecinfer', 'spectral', 'kivi', 'kivi_sink', 'svdq', 'kitty', 'adakv', 'xquant', 'kvquant', 'palu', 'cachegen', 'minicache', 'gear', 'zipcache', 'snapkv', 'streaming_llm', 'h2o', 'tova', 'pyramidkv', 'squeeze', 'chunkkv', 'cam', 'xkv', 'nsnquant', 'knorm', 'skvq', 'qfilters', 'keyformer', 'morphkv', 'kvzip', 'kvtc', 'curdkv', 'nestedkv', 'amc', 'a2ats', 'anchorkv', 'rocketkv', 'age_tiered'])

• head_dim (int)

• bit_width_inlier (int | list)

• bit_width_outlier (int | None)

• jl_dim (int | None)

• n_outlier_channels (int | None)

• n_calib_tokens (int | None)

• enable_vectorized_attend (bool)

• enable_outlier_two_stream (bool)

• enable_fused_query_dot (bool)

• seed (int)

• dtype (Any)

• capacity (int | None)

• sliding_window (int | None)

• store (ArtifactStore | None)

• observers (list)

• options (object | None)

• key_sub_dim (int)

• value_sub_dim (int)

• key_codebook_bits (int)

• value_codebook_bits (int)

• residual_length (int)

• kivi_group_size (int)

• svdq_rank (int | None)

• svdq_energy_threshold (float)

• svdq_bit_schedule (tuple[int, ...])

• svdq_group_size (int)

• kitty_hi_fraction (float)

• kitty_hi_bit (int)

• kitty_lo_bit (int)

• kitty_group_size (int)

• adakv_target_avg_bits (float)

• adakv_lo_bit (int)

• adakv_mid_bit (int)

• adakv_hi_bit (int)

• adakv_group_size (int)

• adakv_update_interval (int)

• adakv_importance (str)

• adakv_obs_window (int)

• xquant_group_size (int)

• xquant_base_bits (int)

• xquant_residual_bits (int)

• xquant_group_quant_size (int)

• xquant_max_ctx (int)

• kvquant_bits (int)

• kvquant_outlier_fraction (float)

• kvquant_group_size (int)

• kvquant_lloyd_iters (int)

• kvquant_refit_interval (int)

• kvquant_n_sink (int)

• palu_rank (int | None)

• palu_energy_threshold (float)

• palu_n_head_groups (int)

• palu_hi_bit (int)

• palu_lo_bit (int)

• palu_hi_fraction (float)

• palu_group_size (int)

• palu_quantize_values (bool)

• cachegen_bits (int)

• cachegen_group_size (int)

• cachegen_use_delta (bool)

• cachegen_per_channel (bool)

• cachegen_layer_groups (int)

• cachegen_resolved_bits (int | None)

• minicache_start_frac (float)

• minicache_group_size (int)

• minicache_retention_threshold (float)

• minicache_slerp_t (float)

• minicache_max_ctx (int)

• gear_bits (int)

• gear_rank (int | None)

• gear_energy_threshold (float)

• gear_sparse_fraction (float)

• gear_group_size (int)

• gear_quantize_values (bool)

• zipcache_hi_bits (int)

• zipcache_lo_bits (int)

• zipcache_hi_fraction (float)

• zipcache_group_size (int)

• zipcache_quantize_values (bool)

• snap_dtype (str)

• snap_batched_scoring (bool)

• snap_backend (str)

• snap_budget (int)

• snap_obs_window (int)

• snap_n_sink (int)

• stream_n_sink (int)

• stream_window_size (int)

• h2o_budget (int)

• h2o_n_sink (int)

• h2o_rope_base (float)

• h2o_grace (int)

• h2o_decay (float)

• tova_budget (int)

• tova_n_sink (int)

• tova_backend (str)

• pyramid_budget (int)

• pyramid_backend (str)

• pyramid_n_sink (int)

• pyramid_beta (float)

• pyramid_resolved_budget (int | None)

• squeeze_budget (int)

• squeeze_n_sink (int)

• squeeze_strength (float)

• squeeze_resolved_budget (int | None)

• chunkkv_budget (int)

• chunkkv_chunk_size (int)

• chunkkv_n_sink (int)

• chunkkv_score (str)

• chunkkv_reuse_layers (int)

• cam_budget (int)

• cam_n_sink (int)

• cam_merge (str)

• cam_merge_keys (bool)

• cam_merge_gate (bool)

• xkv_group_size (int)

• xkv_rank (int | None)

• xkv_energy_threshold (float)

• xkv_latent_bits (int)

• xkv_group_quant_size (int)

• xkv_max_ctx (int)

• nsn_bits (int)

• nsn_residual_length (int)

• nsn_codebook_size (int)

• nsn_subvector_dim (int)

• nsn_seed (int)

• nsn_max_ctx (int)

• knorm_budget (int)

• knorm_n_sink (int)

• knorm_recent (int)

• knorm_keep (str)

• skvq_bits_key (int)

• skvq_bits_value (int)

• skvq_group_size (int)

• skvq_window (int)

• skvq_n_sink (int)

• skvq_reorder (bool)

• skvq_clip_search (bool)

• skvq_clip_alpha (float)

• skvq_max_ctx (int | None)

• qfilters_budget (int)

• qfilters_n_sink (int)

• qfilters_recent (int)

• qfilters_calib_tokens (int)

• qfilters_sign (int)

• qfilters_min_retention (float | None)

• qfilters_on_low_retention (Literal['warn', 'raise', 'ignore'])

• keyformer_budget (int)

• keyformer_n_sink (int)

• keyformer_recent (int)

• keyformer_tau (float | None)

• keyformer_tau_init (float)

• keyformer_tau_end (float)

• keyformer_anneal_steps (int)

• keyformer_rope_base (float)

• keyformer_seed (int)

• morphkv_budget (int)

• morphkv_n_sink (int)

• morphkv_window (int)

• kvzip_budget (int)

• kvzip_n_sink (int)

• kvzip_probe (str)

• kvtc_bit_budget (int)

• kvtc_bit_choices (tuple)

• kvtc_beta (float)

• curdkv_budget (int)

• curdkv_n_sink (int)

• curdkv_rank_cap (int)

• curdkv_rope_base (float)

• nestedkv_budget (int)

• nestedkv_n_sink (int)

• nestedkv_window (int)

• nestedkv_beta (float)

• nestedkv_tau (float)

• nestedkv_kappa (float)

• nestedkv_safeguard_alpha (float)

• amc_k_high (float)

• amc_k_mid (float)

• amc_use_query_saliency (bool)

• amc_query_alpha (float)

• amc_adaptive_thresholds (bool)

• amc_threshold_window (int)

• amc_gamma (float)

• amc_calib_variance (float | None)

• amc_group_size (int)

• a2ats_codebook_bits (int)

• a2ats_sub_dim (int)

• a2ats_window (int)

• a2ats_b (int)

• a2ats_use_query_aware (bool)

• a2ats_beta (float)

• a2ats_retrieval_fraction (float)

• a2ats_rope_base (float)

• a2ats_rope_scaling (Any)

• a2ats_query_h (Any)

• a2ats_codebook (Any)

• anchorkv_theta (float)

• anchorkv_window (int)

• anchorkv_rho (float)

• anchorkv_anchor_frac (float)

• anchorkv_residual_bits (int)

• anchorkv_seed (int)

• rocketkv_compression_ratio (float)

• rocketkv_page_size (int | None)

• rocketkv_head_topk1 (int | None)

• rocketkv_obs_window (int)

• rocketkv_n_sink (int)

• age_recent_boundary (int)

• age_mid_boundary (int)

• age_bits_recent (int)

• age_bits_mid (int)

• age_bits_old (int)

• age_group_size (int)

• n_sink_tokens (int)

• smooth_factors (Any)

• key_codebook (Any)

• value_codebook (Any)

• spectral_key_d_eff (int)

• spectral_val_d_eff (int)

• spectral_apply_qjl (bool)

• spectral_model_name (str)

• use_metal_kernels (bool | None)

• fused_sdpa (bool | None)

• fused_sdpa_max_ctx (int | None)

• fused_sdpa_memory_bound (bool)

method: Literal['turboquant_prod', 'turboquant_mse', 'turboquant_rvq', 'polar', 'qjl', 'vecinfer', 'spectral', 'kivi', 'kivi_sink', 'svdq', 'kitty', 'adakv', 'xquant', 'kvquant', 'palu', 'cachegen', 'minicache', 'gear', 'zipcache', 'snapkv', 'streaming_llm', 'h2o', 'tova', 'pyramidkv', 'squeeze', 'chunkkv', 'cam', 'xkv', 'nsnquant', 'knorm', 'skvq', 'qfilters', 'keyformer', 'morphkv', 'kvzip', 'kvtc', 'curdkv', 'nestedkv', 'amc', 'a2ats', 'anchorkv', 'rocketkv', 'age_tiered'] = 'turboquant_rvq'

head_dim: int = 128

bit_width_inlier: int | list = 2

bit_width_outlier: int | None = None

jl_dim: int | None = None

n_outlier_channels: int | None = None

n_calib_tokens: int | None = None

enable_vectorized_attend: bool = True

enable_outlier_two_stream: bool = False

enable_fused_query_dot: bool = False

seed: int = 42

dtype: Any = None

capacity: int | None = None

sliding_window: int | None = None

store: ArtifactStore | None = None

observers: list

options: object | None = None

key_sub_dim: int = 4

value_sub_dim: int = 8

key_codebook_bits: int = 12

value_codebook_bits: int = 8

residual_length: int = 128

kivi_group_size: int = 32

svdq_rank: int | None = None

svdq_energy_threshold: float = 0.95

svdq_bit_schedule: tuple[int, ...] = (8, 4, 2, 1, 1, 0, 0, 0)

svdq_group_size: int = 32

kitty_hi_fraction: float = 0.25

kitty_hi_bit: int = 4

kitty_lo_bit: int = 2

kitty_group_size: int = 32

adakv_target_avg_bits: float = 2.5

adakv_lo_bit: int = 2

adakv_mid_bit: int = 3

adakv_hi_bit: int = 4

adakv_group_size: int = 32

adakv_update_interval: int = 1

adakv_importance: str = 'norm_variance'

adakv_obs_window: int = 32

xquant_group_size: int = 2

xquant_base_bits: int = 2

xquant_residual_bits: int = 4

xquant_group_quant_size: int = 32

xquant_max_ctx: int = 8192

kvquant_bits: int = 3

kvquant_outlier_fraction: float = 0.01

kvquant_group_size: int = 32

kvquant_lloyd_iters: int = 8

kvquant_refit_interval: int = 0

kvquant_n_sink: int = 1

palu_rank: int | None = None

palu_energy_threshold: float = 0.9

palu_n_head_groups: int = 4

palu_hi_bit: int = 4

palu_lo_bit: int = 2

palu_hi_fraction: float = 0.25

palu_group_size: int = 32

palu_quantize_values: bool = True

cachegen_bits: int = 4

cachegen_group_size: int = 32

cachegen_use_delta: bool = True

cachegen_per_channel: bool = True

cachegen_layer_groups: int = 3

cachegen_resolved_bits: int | None = None

minicache_start_frac: float = 0.5

minicache_group_size: int = 2

minicache_retention_threshold: float = 0.9

minicache_slerp_t: float = 0.5

minicache_max_ctx: int = 8192

gear_bits: int = 2

gear_rank: int | None = None

gear_energy_threshold: float = 0.9

gear_sparse_fraction: float = 0.01

gear_group_size: int = 32

gear_quantize_values: bool = True

zipcache_hi_bits: int = 4

zipcache_lo_bits: int = 2

zipcache_hi_fraction: float = 0.2

zipcache_group_size: int = 32

zipcache_quantize_values: bool = True

snap_dtype: str = 'auto'

snap_batched_scoring: bool = True

snap_backend: str = 'auto'

snap_budget: int = 512

snap_obs_window: int = 32

snap_n_sink: int = 4

stream_n_sink: int = 4

stream_window_size: int = 512

h2o_budget: int = 512

h2o_n_sink: int = 4

h2o_rope_base: float = 10000.0

h2o_grace: int = 16

h2o_decay: float = 0.98

tova_budget: int = 512

tova_n_sink: int = 4

tova_backend: str = 'auto'

pyramid_budget: int = 512

pyramid_backend: str = 'reference'

pyramid_n_sink: int = 4

pyramid_beta: float = 2.0

pyramid_resolved_budget: int | None = None

squeeze_budget: int = 512

squeeze_n_sink: int = 4

squeeze_strength: float = 1.0

squeeze_resolved_budget: int | None = None

chunkkv_budget: int = 512

chunkkv_chunk_size: int = 8

chunkkv_n_sink: int = 4

chunkkv_score: str = 'attn_mass'

chunkkv_reuse_layers: int = 1

cam_budget: int = 512

cam_n_sink: int = 4

cam_merge: str = 'sim_weighted'

cam_merge_keys: bool = False

cam_merge_gate: bool = True

xkv_group_size: int = 2

xkv_rank: int | None = None

xkv_energy_threshold: float = 0.95

xkv_latent_bits: int = 4

xkv_group_quant_size: int = 32

xkv_max_ctx: int = 8192

nsn_bits: int = 2

nsn_residual_length: int = 64

nsn_codebook_size: int = 256

nsn_subvector_dim: int = 8

nsn_seed: int = 1234

nsn_max_ctx: int = 8192

knorm_budget: int = 512

knorm_n_sink: int = 4

knorm_recent: int = 0

knorm_keep: str = 'low'

skvq_bits_key: int = 2

skvq_bits_value: int = 2

skvq_group_size: int = 32

skvq_window: int = 128

skvq_n_sink: int = 5

skvq_reorder: bool = True

skvq_clip_search: bool = True

skvq_clip_alpha: float = 1.0

skvq_max_ctx: int | None = None

qfilters_budget: int = 512

qfilters_n_sink: int = 4

qfilters_recent: int = 0

qfilters_calib_tokens: int = 128

qfilters_sign: int = 1

qfilters_min_retention: float | None = 0.9

qfilters_on_low_retention: Literal['warn', 'raise', 'ignore'] = 'warn'

keyformer_budget: int = 512

keyformer_n_sink: int = 4

keyformer_recent: int = 0

keyformer_tau: float | None = None

keyformer_tau_init: float = 1.0

keyformer_tau_end: float = 1.0

keyformer_anneal_steps: int = 0

keyformer_rope_base: float = 10000.0

keyformer_seed: int = 0

morphkv_budget: int = 512

morphkv_n_sink: int = 4

morphkv_window: int = 8

kvzip_budget: int = 512

kvzip_n_sink: int = 4

kvzip_probe: str = 'context'

kvtc_bit_budget: int = 512

kvtc_bit_choices: tuple = (0, 1, 2, 3, 4, 6, 8)

kvtc_beta: float = 3.5

curdkv_budget: int = 512

curdkv_n_sink: int = 4

curdkv_rank_cap: int = 16

curdkv_rope_base: float = 10000.0

nestedkv_budget: int = 512

nestedkv_n_sink: int = 4

nestedkv_window: int = 64

nestedkv_beta: float = 3.0

nestedkv_tau: float = 0.6

nestedkv_kappa: float = 10.0

nestedkv_safeguard_alpha: float = 0.2

amc_k_high: float = 0.2

amc_k_mid: float = 0.3

amc_use_query_saliency: bool = False

amc_query_alpha: float = 0.5

amc_adaptive_thresholds: bool = False

amc_threshold_window: int = 64

amc_gamma: float = 0.1

amc_calib_variance: float | None = None

amc_group_size: int = 32

a2ats_codebook_bits: int = 8

a2ats_sub_dim: int = 8

a2ats_window: int = 128

a2ats_b: int = 2048

a2ats_use_query_aware: bool = True

a2ats_beta: float = 0.5

a2ats_retrieval_fraction: float = 0.2

a2ats_rope_base: float = 10000.0

a2ats_rope_scaling: Any = None

a2ats_query_h: Any = None

a2ats_codebook: Any = None

anchorkv_theta: float = 0.05

anchorkv_window: int = 32

anchorkv_rho: float = 0.7

anchorkv_anchor_frac: float = 0.0078125

anchorkv_residual_bits: int = 2

anchorkv_seed: int = 42

rocketkv_compression_ratio: float = 8.0

rocketkv_page_size: int | None = None

rocketkv_head_topk1: int | None = None

rocketkv_obs_window: int = 32

rocketkv_n_sink: int = 4

age_recent_boundary: int = 128

age_mid_boundary: int = 1024

age_bits_recent: int = 8

age_bits_mid: int = 4

age_bits_old: int = 2

age_group_size: int = 32

n_sink_tokens: int = 5

smooth_factors: Any = None

key_codebook: Any = None

value_codebook: Any = None

spectral_key_d_eff: int = 4

spectral_val_d_eff: int = 50

spectral_apply_qjl: bool = True

spectral_model_name: str = 'model'

use_metal_kernels: bool | None = None

fused_sdpa: bool | None = False

fused_sdpa_max_ctx: int | None = None

fused_sdpa_memory_bound: bool = False

class veloxquant_mlx.cache.base.KVCacheFactory

Bases: object

Factory for creating KVCache instances from a KVCacheConfig.

static create(config)

Instantiate a KVCache from the given configuration.

Parameters:

config (KVCacheConfig) – KVCacheConfig instance.

Return type:

KVCache | _MLXKVCache

Returns:

Configured KVCache. The concrete type depends on config.method – STANDALONE_METHODS produce VeloxQuant’s own KVCache (core. abstractions), every other method produces an mlx_lm.models. cache.KVCache subclass — see the STANDALONE_METHODS comment above.

class veloxquant_mlx.cache.base.KVCacheBuilder

Bases: object

Fluent builder for KVCache construction with validation.

Example:

cache = (
    KVCacheBuilder()
    .with_method("turboquant_prod")
    .with_head_dim(128)
    .with_bit_width(inlier=2, outlier=3)
    .with_jl_dim(128)
    .with_seed(42)
    .build()
)

with_method(method)

Set the quantisation method.

Parameters:

method (str) – One of ‘turboquant_prod’, ‘turboquant_mse’, ‘polar’, ‘qjl’.

Return type:

KVCacheBuilder

with_head_dim(d)

Set the attention head dimension.

Parameters:

d (int) – Head dimension (must be a power of 2).

Return type:

KVCacheBuilder

with_bit_width(inlier, outlier=None)

Set bit-width(s).

Parameters:

• inlier – Bit-width for inlier channels. Either an int (uniform across all layers) or a list[int] of length n_layers for RateQuant-style per-layer allocation. When a list is supplied, this builder must be consumed via KVCacheBuilder.for_model(model, config); direct .build() rejects the list.

• outlier (int | None) – Bit-width for outlier channels (defaults to inlier if None).

Return type:

KVCacheBuilder

with_jl_dim(m)

Set the JL projection dimension.

Parameters:

m (int) – Must be <= head_dim.

Return type:

KVCacheBuilder

with_n_outlier_channels(n)

Set the number of outlier channels to detect.

Parameters:

n (int) – Must be < head_dim.

Return type:

KVCacheBuilder

with_n_calib_tokens(n)

Set calibration token count for outlier activation.

Return type:

KVCacheBuilder

Parameters:

n (int)

with_vectorized_attend(enabled=True)

Enable vectorized packed-key unpack in attend().

Return type:

KVCacheBuilder

Parameters:

enabled (bool)

with_outlier_two_stream(enabled=True)

Enable outlier/inlier split cache after calibration.

Return type:

KVCacheBuilder

Parameters:

enabled (bool)

with_fused_query_dot(enabled=True)

Enable fused rotated-query + codebook-dot path.

Return type:

KVCacheBuilder

Parameters:

enabled (bool)

with_seed(seed)

Set the random seed.

Parameters:

seed (int) – Integer seed.

Return type:

KVCacheBuilder

with_precision(dtype)

Set the compute dtype.

Parameters:

dtype (Any) – MLX dtype (e.g. mx.float16).

Return type:

KVCacheBuilder

with_capacity(max_tokens)

Set the maximum number of tokens to store.

Parameters:

max_tokens (int) – Positive integer.

Return type:

KVCacheBuilder

with_artifact_store(store)

Provide an ArtifactStore for loading precomputed artifacts.

Parameters:

store (ArtifactStore) – ArtifactStore instance.

Return type:

KVCacheBuilder

with_observer(observer)

Attach a QuantizationObserver.

Parameters:

observer (QuantizationObserver) – Observer instance.

Return type:

KVCacheBuilder

with_sliding_window(window_size)

Wrap the cache with a sliding-window eviction policy.

Parameters:

window_size (int) – Number of tokens to keep.

Return type:

KVCacheBuilder

build()

Validate the configuration and construct the KVCache.

Return type:

KVCache | _MLXKVCache

Returns:

Configured KVCache instance.

Raises:

QuantizerConfigError – If any validation check fails.

static for_model(model, config)

Build one KVCache per language-model layer, sized per-layer.

Works for text-only and VLM models (Qwen2-VL, Qwen3-VL, Mistral, etc.). Layers without a self_attn attribute (MoE gates, etc.) fall back to a standard fp16 KVCache so the list length always matches model.layers.

Per-layer bit-widths (RateQuant)

If config.bit_width_inlier is a list[int], element i is used for layer i. The list length must equal the number of attention-bearing layers (layers without self_attn are skipped from the count). This lets RateQuant-style mixed-precision allocations be passed through the standard API without manual cache wiring.

type model:

param model:

Loaded mlx_lm model instance.

type config:

KVCacheConfig

param config:

KVCacheConfig specifying method, bit_width_inlier, seed, etc. head_dim is overridden per-layer.

rtype:

list

returns:

List of KVCache instances, one per language-model layer.

raises QuantizerConfigError:

If config.method is a standalone method (see STANDALONE_METHODS) that does not implement the mlx_lm KVCache serving contract.

Parameters:

config (KVCacheConfig)

Return type:

list