veloxquant_mlx package

veloxquant_mlx — KV cache quantization for Apple Silicon MLX.

Implements TurboQuant, TurboQuantRVQ, PolarQuant, and QJL plus the RateQuant per-layer bit allocator for production LLM inference.

Preview APIs: AutoOptimizer (automatic hardware-aware strategy selection, RFC #469) and CacheRoutePlanner (rate-aware session routing) are exported here for convenience but are subject to change without a major version bump — see their module docstrings (veloxquant_mlx.planning, veloxquant_mlx.routing.cacheroute) for what “preview” means for each.

class veloxquant_mlx.KVCacheBuilder

Bases: object

Fluent builder for KVCache construction with validation.

Example:

cache = (
    KVCacheBuilder()
    .with_method("turboquant_prod")
    .with_head_dim(128)
    .with_bit_width(inlier=2, outlier=3)
    .with_jl_dim(128)
    .with_seed(42)
    .build()
)

with_method(method)

Set the quantisation method.

Parameters:

method (str) – One of ‘turboquant_prod’, ‘turboquant_mse’, ‘polar’, ‘qjl’.

Return type:

KVCacheBuilder

with_head_dim(d)

Set the attention head dimension.

Parameters:

d (int) – Head dimension (must be a power of 2).

Return type:

KVCacheBuilder

with_bit_width(inlier, outlier=None)

Set bit-width(s).

Parameters:

• inlier – Bit-width for inlier channels. Either an int (uniform across all layers) or a list[int] of length n_layers for RateQuant-style per-layer allocation. When a list is supplied, this builder must be consumed via KVCacheBuilder.for_model(model, config); direct .build() rejects the list.

• outlier (int | None) – Bit-width for outlier channels (defaults to inlier if None).

Return type:

KVCacheBuilder

with_jl_dim(m)

Set the JL projection dimension.

Parameters:

m (int) – Must be <= head_dim.

Return type:

KVCacheBuilder

with_n_outlier_channels(n)

Set the number of outlier channels to detect.

Parameters:

n (int) – Must be < head_dim.

Return type:

KVCacheBuilder

with_n_calib_tokens(n)

Set calibration token count for outlier activation.

Return type:

KVCacheBuilder

Parameters:

n (int)

with_vectorized_attend(enabled=True)

Enable vectorized packed-key unpack in attend().

Return type:

KVCacheBuilder

Parameters:

enabled (bool)

with_outlier_two_stream(enabled=True)

Enable outlier/inlier split cache after calibration.

Return type:

KVCacheBuilder

Parameters:

enabled (bool)

with_fused_query_dot(enabled=True)

Enable fused rotated-query + codebook-dot path.

Return type:

KVCacheBuilder

Parameters:

enabled (bool)

with_seed(seed)

Set the random seed.

Parameters:

seed (int) – Integer seed.

Return type:

KVCacheBuilder

with_precision(dtype)

Set the compute dtype.

Parameters:

dtype (Any) – MLX dtype (e.g. mx.float16).

Return type:

KVCacheBuilder

with_capacity(max_tokens)

Set the maximum number of tokens to store.

Parameters:

max_tokens (int) – Positive integer.

Return type:

KVCacheBuilder

with_artifact_store(store)

Provide an ArtifactStore for loading precomputed artifacts.

Parameters:

store (ArtifactStore) – ArtifactStore instance.

Return type:

KVCacheBuilder

with_observer(observer)

Attach a QuantizationObserver.

Parameters:

observer (QuantizationObserver) – Observer instance.

Return type:

KVCacheBuilder

with_sliding_window(window_size)

Wrap the cache with a sliding-window eviction policy.

Parameters:

window_size (int) – Number of tokens to keep.

Return type:

KVCacheBuilder

build()

Validate the configuration and construct the KVCache.

Return type:

KVCache | _MLXKVCache

Returns:

Configured KVCache instance.

Raises:

QuantizerConfigError – If any validation check fails.

static for_model(model, config)

Build one KVCache per language-model layer, sized per-layer.

Works for text-only and VLM models (Qwen2-VL, Qwen3-VL, Mistral, etc.). Layers without a self_attn attribute (MoE gates, etc.) fall back to a standard fp16 KVCache so the list length always matches model.layers.

Per-layer bit-widths (RateQuant)

If config.bit_width_inlier is a list[int], element i is used for layer i. The list length must equal the number of attention-bearing layers (layers without self_attn are skipped from the count). This lets RateQuant-style mixed-precision allocations be passed through the standard API without manual cache wiring.

type model:

param model:

Loaded mlx_lm model instance.

type config:

KVCacheConfig

param config:

KVCacheConfig specifying method, bit_width_inlier, seed, etc. head_dim is overridden per-layer.

rtype:

list

returns:

List of KVCache instances, one per language-model layer.

raises QuantizerConfigError:

If config.method is a standalone method (see STANDALONE_METHODS) that does not implement the mlx_lm KVCache serving contract.

Parameters:

config (KVCacheConfig)

Return type:

list

class veloxquant_mlx.KVCacheConfig(method='turboquant_rvq', head_dim=128, bit_width_inlier=2, bit_width_outlier=None, jl_dim=None, n_outlier_channels=None, n_calib_tokens=None, enable_vectorized_attend=True, enable_outlier_two_stream=False, enable_fused_query_dot=False, seed=42, dtype=None, capacity=None, sliding_window=None, store=None, observers=<factory>, options=None, key_sub_dim=4, value_sub_dim=8, key_codebook_bits=12, value_codebook_bits=8, residual_length=128, kivi_group_size=32, svdq_rank=None, svdq_energy_threshold=0.95, svdq_bit_schedule=(8, 4, 2, 1, 1, 0, 0, 0), svdq_group_size=32, kitty_hi_fraction=0.25, kitty_hi_bit=4, kitty_lo_bit=2, kitty_group_size=32, adakv_target_avg_bits=2.5, adakv_lo_bit=2, adakv_mid_bit=3, adakv_hi_bit=4, adakv_group_size=32, adakv_update_interval=1, adakv_importance='norm_variance', adakv_obs_window=32, xquant_group_size=2, xquant_base_bits=2, xquant_residual_bits=4, xquant_group_quant_size=32, xquant_max_ctx=8192, kvquant_bits=3, kvquant_outlier_fraction=0.01, kvquant_group_size=32, kvquant_lloyd_iters=8, kvquant_refit_interval=0, kvquant_n_sink=1, palu_rank=None, palu_energy_threshold=0.9, palu_n_head_groups=4, palu_hi_bit=4, palu_lo_bit=2, palu_hi_fraction=0.25, palu_group_size=32, palu_quantize_values=True, cachegen_bits=4, cachegen_group_size=32, cachegen_use_delta=True, cachegen_per_channel=True, cachegen_layer_groups=3, cachegen_resolved_bits=None, minicache_start_frac=0.5, minicache_group_size=2, minicache_retention_threshold=0.9, minicache_slerp_t=0.5, minicache_max_ctx=8192, gear_bits=2, gear_rank=None, gear_energy_threshold=0.9, gear_sparse_fraction=0.01, gear_group_size=32, gear_quantize_values=True, zipcache_hi_bits=4, zipcache_lo_bits=2, zipcache_hi_fraction=0.2, zipcache_group_size=32, zipcache_quantize_values=True, snap_dtype='auto', snap_batched_scoring=True, snap_backend='auto', snap_budget=512, snap_obs_window=32, snap_n_sink=4, stream_n_sink=4, stream_window_size=512, h2o_budget=512, h2o_n_sink=4, h2o_rope_base=10000.0, h2o_grace=16, h2o_decay=0.98, tova_budget=512, tova_n_sink=4, tova_backend='auto', pyramid_budget=512, pyramid_backend='reference', pyramid_n_sink=4, pyramid_beta=2.0, pyramid_resolved_budget=None, squeeze_budget=512, squeeze_n_sink=4, squeeze_strength=1.0, squeeze_resolved_budget=None, chunkkv_budget=512, chunkkv_chunk_size=8, chunkkv_n_sink=4, chunkkv_score='attn_mass', chunkkv_reuse_layers=1, cam_budget=512, cam_n_sink=4, cam_merge='sim_weighted', cam_merge_keys=False, cam_merge_gate=True, xkv_group_size=2, xkv_rank=None, xkv_energy_threshold=0.95, xkv_latent_bits=4, xkv_group_quant_size=32, xkv_max_ctx=8192, nsn_bits=2, nsn_residual_length=64, nsn_codebook_size=256, nsn_subvector_dim=8, nsn_seed=1234, nsn_max_ctx=8192, knorm_budget=512, knorm_n_sink=4, knorm_recent=0, knorm_keep='low', skvq_bits_key=2, skvq_bits_value=2, skvq_group_size=32, skvq_window=128, skvq_n_sink=5, skvq_reorder=True, skvq_clip_search=True, skvq_clip_alpha=1.0, skvq_max_ctx=None, qfilters_budget=512, qfilters_n_sink=4, qfilters_recent=0, qfilters_calib_tokens=128, qfilters_sign=1, qfilters_min_retention=0.9, qfilters_on_low_retention='warn', keyformer_budget=512, keyformer_n_sink=4, keyformer_recent=0, keyformer_tau=None, keyformer_tau_init=1.0, keyformer_tau_end=1.0, keyformer_anneal_steps=0, keyformer_rope_base=10000.0, keyformer_seed=0, morphkv_budget=512, morphkv_n_sink=4, morphkv_window=8, kvzip_budget=512, kvzip_n_sink=4, kvzip_probe='context', kvtc_bit_budget=512, kvtc_bit_choices=(0, 1, 2, 3, 4, 6, 8), kvtc_beta=3.5, curdkv_budget=512, curdkv_n_sink=4, curdkv_rank_cap=16, curdkv_rope_base=10000.0, nestedkv_budget=512, nestedkv_n_sink=4, nestedkv_window=64, nestedkv_beta=3.0, nestedkv_tau=0.6, nestedkv_kappa=10.0, nestedkv_safeguard_alpha=0.2, amc_k_high=0.2, amc_k_mid=0.3, amc_use_query_saliency=False, amc_query_alpha=0.5, amc_adaptive_thresholds=False, amc_threshold_window=64, amc_gamma=0.1, amc_calib_variance=None, amc_group_size=32, a2ats_codebook_bits=8, a2ats_sub_dim=8, a2ats_window=128, a2ats_b=2048, a2ats_use_query_aware=True, a2ats_beta=0.5, a2ats_retrieval_fraction=0.2, a2ats_rope_base=10000.0, a2ats_rope_scaling=None, a2ats_query_h=None, a2ats_codebook=None, anchorkv_theta=0.05, anchorkv_window=32, anchorkv_rho=0.7, anchorkv_anchor_frac=0.0078125, anchorkv_residual_bits=2, anchorkv_seed=42, rocketkv_compression_ratio=8.0, rocketkv_page_size=None, rocketkv_head_topk1=None, rocketkv_obs_window=32, rocketkv_n_sink=4, age_recent_boundary=128, age_mid_boundary=1024, age_bits_recent=8, age_bits_mid=4, age_bits_old=2, age_group_size=32, n_sink_tokens=5, smooth_factors=None, key_codebook=None, value_codebook=None, spectral_key_d_eff=4, spectral_val_d_eff=50, spectral_apply_qjl=True, spectral_model_name='model', use_metal_kernels=None, fused_sdpa=False, fused_sdpa_max_ctx=None, fused_sdpa_memory_bound=False)

Bases: object

Configuration for a KVCache instance.

Variables:

• method – Quantisation algorithm.

• head_dim – Attention head dimension (d).

• bit_width_inlier – Bit-width for inlier channels. Either a single int applied uniformly across all layers, OR a list[int] of length n_layers for per-layer RateQuant-style allocation. When passed as a list, KVCacheBuilder.for_model() consumes element i for layer i; KVCacheFactory.create() requires an int.

• bit_width_outlier – Bit-width for outlier channels (None → same as inlier).

• jl_dim – JL projection dimension m. None means m = head_dim for QJL, which is cheap but noisy (score correlation ~0.64 at d=128 vs ~0.88 at m=4d); raise it when attention quality matters. See #774.

• n_outlier_channels – Number of outlier channels to detect.

• seed – Random seed.

• dtype – MLX dtype for computations.

• capacity – Maximum tokens to store (None → unlimited).

• sliding_window – If set, wrap cache with sliding-window eviction.

• store – ArtifactStore to load precomputed artifacts from.

• observers – List of QuantizationObserver instances.

• options – Method-specific option dataclass from veloxquant_mlx.cache.options (e.g. H2OOptions, SnapKVOptions), grouping the flat per-method fields below (h2o_budget, snap_budget, …) into one small object per method. Preferred over setting the flat fields directly (#420): values from options populate the matching flat fields at construction time (see __post_init__), so every existing cache class – which reads the flat fields via getattr(config, "h2o_budget", 512) – keeps working unchanged. Setting a flat per-method field directly still works but emits a DeprecationWarning naming its options replacement.

Parameters:

• method (Literal['turboquant_prod', 'turboquant_mse', 'turboquant_rvq', 'polar', 'qjl', 'vecinfer', 'spectral', 'kivi', 'kivi_sink', 'svdq', 'kitty', 'adakv', 'xquant', 'kvquant', 'palu', 'cachegen', 'minicache', 'gear', 'zipcache', 'snapkv', 'streaming_llm', 'h2o', 'tova', 'pyramidkv', 'squeeze', 'chunkkv', 'cam', 'xkv', 'nsnquant', 'knorm', 'skvq', 'qfilters', 'keyformer', 'morphkv', 'kvzip', 'kvtc', 'curdkv', 'nestedkv', 'amc', 'a2ats', 'anchorkv', 'rocketkv', 'age_tiered'])

• head_dim (int)

• bit_width_inlier (int | list)

• bit_width_outlier (int | None)

• jl_dim (int | None)

• n_outlier_channels (int | None)

• n_calib_tokens (int | None)

• enable_vectorized_attend (bool)

• enable_outlier_two_stream (bool)

• enable_fused_query_dot (bool)

• seed (int)

• dtype (Any)

• capacity (int | None)

• sliding_window (int | None)

• store (ArtifactStore | None)

• observers (list)

• options (object | None)

• key_sub_dim (int)

• value_sub_dim (int)

• key_codebook_bits (int)

• value_codebook_bits (int)

• residual_length (int)

• kivi_group_size (int)

• svdq_rank (int | None)

• svdq_energy_threshold (float)

• svdq_bit_schedule (tuple[int, ...])

• svdq_group_size (int)

• kitty_hi_fraction (float)

• kitty_hi_bit (int)

• kitty_lo_bit (int)

• kitty_group_size (int)

• adakv_target_avg_bits (float)

• adakv_lo_bit (int)

• adakv_mid_bit (int)

• adakv_hi_bit (int)

• adakv_group_size (int)

• adakv_update_interval (int)

• adakv_importance (str)

• adakv_obs_window (int)

• xquant_group_size (int)

• xquant_base_bits (int)

• xquant_residual_bits (int)

• xquant_group_quant_size (int)

• xquant_max_ctx (int)

• kvquant_bits (int)

• kvquant_outlier_fraction (float)

• kvquant_group_size (int)

• kvquant_lloyd_iters (int)

• kvquant_refit_interval (int)

• kvquant_n_sink (int)

• palu_rank (int | None)

• palu_energy_threshold (float)

• palu_n_head_groups (int)

• palu_hi_bit (int)

• palu_lo_bit (int)

• palu_hi_fraction (float)

• palu_group_size (int)

• palu_quantize_values (bool)

• cachegen_bits (int)

• cachegen_group_size (int)

• cachegen_use_delta (bool)

• cachegen_per_channel (bool)

• cachegen_layer_groups (int)

• cachegen_resolved_bits (int | None)

• minicache_start_frac (float)

• minicache_group_size (int)

• minicache_retention_threshold (float)

• minicache_slerp_t (float)

• minicache_max_ctx (int)

• gear_bits (int)

• gear_rank (int | None)

• gear_energy_threshold (float)

• gear_sparse_fraction (float)

• gear_group_size (int)

• gear_quantize_values (bool)

• zipcache_hi_bits (int)

• zipcache_lo_bits (int)

• zipcache_hi_fraction (float)

• zipcache_group_size (int)

• zipcache_quantize_values (bool)

• snap_dtype (str)

• snap_batched_scoring (bool)

• snap_backend (str)

• snap_budget (int)

• snap_obs_window (int)

• snap_n_sink (int)

• stream_n_sink (int)

• stream_window_size (int)

• h2o_budget (int)

• h2o_n_sink (int)

• h2o_rope_base (float)

• h2o_grace (int)

• h2o_decay (float)

• tova_budget (int)

• tova_n_sink (int)

• tova_backend (str)

• pyramid_budget (int)

• pyramid_backend (str)

• pyramid_n_sink (int)

• pyramid_beta (float)

• pyramid_resolved_budget (int | None)

• squeeze_budget (int)

• squeeze_n_sink (int)

• squeeze_strength (float)

• squeeze_resolved_budget (int | None)

• chunkkv_budget (int)

• chunkkv_chunk_size (int)

• chunkkv_n_sink (int)

• chunkkv_score (str)

• chunkkv_reuse_layers (int)

• cam_budget (int)

• cam_n_sink (int)

• cam_merge (str)

• cam_merge_keys (bool)

• cam_merge_gate (bool)

• xkv_group_size (int)

• xkv_rank (int | None)

• xkv_energy_threshold (float)

• xkv_latent_bits (int)

• xkv_group_quant_size (int)

• xkv_max_ctx (int)

• nsn_bits (int)

• nsn_residual_length (int)

• nsn_codebook_size (int)

• nsn_subvector_dim (int)

• nsn_seed (int)

• nsn_max_ctx (int)

• knorm_budget (int)

• knorm_n_sink (int)

• knorm_recent (int)

• knorm_keep (str)

• skvq_bits_key (int)

• skvq_bits_value (int)

• skvq_group_size (int)

• skvq_window (int)

• skvq_n_sink (int)

• skvq_reorder (bool)

• skvq_clip_search (bool)

• skvq_clip_alpha (float)

• skvq_max_ctx (int | None)

• qfilters_budget (int)

• qfilters_n_sink (int)

• qfilters_recent (int)

• qfilters_calib_tokens (int)

• qfilters_sign (int)

• qfilters_min_retention (float | None)

• qfilters_on_low_retention (Literal['warn', 'raise', 'ignore'])

• keyformer_budget (int)

• keyformer_n_sink (int)

• keyformer_recent (int)

• keyformer_tau (float | None)

• keyformer_tau_init (float)

• keyformer_tau_end (float)

• keyformer_anneal_steps (int)

• keyformer_rope_base (float)

• keyformer_seed (int)

• morphkv_budget (int)

• morphkv_n_sink (int)

• morphkv_window (int)

• kvzip_budget (int)

• kvzip_n_sink (int)

• kvzip_probe (str)

• kvtc_bit_budget (int)

• kvtc_bit_choices (tuple)

• kvtc_beta (float)

• curdkv_budget (int)

• curdkv_n_sink (int)

• curdkv_rank_cap (int)

• curdkv_rope_base (float)

• nestedkv_budget (int)

• nestedkv_n_sink (int)

• nestedkv_window (int)

• nestedkv_beta (float)

• nestedkv_tau (float)

• nestedkv_kappa (float)

• nestedkv_safeguard_alpha (float)

• amc_k_high (float)

• amc_k_mid (float)

• amc_use_query_saliency (bool)

• amc_query_alpha (float)

• amc_adaptive_thresholds (bool)

• amc_threshold_window (int)

• amc_gamma (float)

• amc_calib_variance (float | None)

• amc_group_size (int)

• a2ats_codebook_bits (int)

• a2ats_sub_dim (int)

• a2ats_window (int)

• a2ats_b (int)

• a2ats_use_query_aware (bool)

• a2ats_beta (float)

• a2ats_retrieval_fraction (float)

• a2ats_rope_base (float)

• a2ats_rope_scaling (Any)

• a2ats_query_h (Any)

• a2ats_codebook (Any)

• anchorkv_theta (float)

• anchorkv_window (int)

• anchorkv_rho (float)

• anchorkv_anchor_frac (float)

• anchorkv_residual_bits (int)

• anchorkv_seed (int)

• rocketkv_compression_ratio (float)

• rocketkv_page_size (int | None)

• rocketkv_head_topk1 (int | None)

• rocketkv_obs_window (int)

• rocketkv_n_sink (int)

• age_recent_boundary (int)

• age_mid_boundary (int)

• age_bits_recent (int)

• age_bits_mid (int)

• age_bits_old (int)

• age_group_size (int)

• n_sink_tokens (int)

• smooth_factors (Any)

• key_codebook (Any)

• value_codebook (Any)

• spectral_key_d_eff (int)

• spectral_val_d_eff (int)

• spectral_apply_qjl (bool)

• spectral_model_name (str)

• use_metal_kernels (bool | None)

• fused_sdpa (bool | None)

• fused_sdpa_max_ctx (int | None)

• fused_sdpa_memory_bound (bool)

method: Literal['turboquant_prod', 'turboquant_mse', 'turboquant_rvq', 'polar', 'qjl', 'vecinfer', 'spectral', 'kivi', 'kivi_sink', 'svdq', 'kitty', 'adakv', 'xquant', 'kvquant', 'palu', 'cachegen', 'minicache', 'gear', 'zipcache', 'snapkv', 'streaming_llm', 'h2o', 'tova', 'pyramidkv', 'squeeze', 'chunkkv', 'cam', 'xkv', 'nsnquant', 'knorm', 'skvq', 'qfilters', 'keyformer', 'morphkv', 'kvzip', 'kvtc', 'curdkv', 'nestedkv', 'amc', 'a2ats', 'anchorkv', 'rocketkv', 'age_tiered'] = 'turboquant_rvq'

head_dim: int = 128

bit_width_inlier: int | list = 2

bit_width_outlier: int | None = None

jl_dim: int | None = None

n_outlier_channels: int | None = None

n_calib_tokens: int | None = None

enable_vectorized_attend: bool = True

enable_outlier_two_stream: bool = False

enable_fused_query_dot: bool = False

seed: int = 42

dtype: Any = None

capacity: int | None = None

sliding_window: int | None = None

store: ArtifactStore | None = None

observers: list

options: object | None = None

key_sub_dim: int = 4

value_sub_dim: int = 8

key_codebook_bits: int = 12

value_codebook_bits: int = 8

residual_length: int = 128

kivi_group_size: int = 32

svdq_rank: int | None = None

svdq_energy_threshold: float = 0.95

svdq_bit_schedule: tuple[int, ...] = (8, 4, 2, 1, 1, 0, 0, 0)

svdq_group_size: int = 32

kitty_hi_fraction: float = 0.25

kitty_hi_bit: int = 4

kitty_lo_bit: int = 2

kitty_group_size: int = 32

adakv_target_avg_bits: float = 2.5

adakv_lo_bit: int = 2

adakv_mid_bit: int = 3

adakv_hi_bit: int = 4

adakv_group_size: int = 32

adakv_update_interval: int = 1

adakv_importance: str = 'norm_variance'

adakv_obs_window: int = 32

xquant_group_size: int = 2

xquant_base_bits: int = 2

xquant_residual_bits: int = 4

xquant_group_quant_size: int = 32

xquant_max_ctx: int = 8192

kvquant_bits: int = 3

kvquant_outlier_fraction: float = 0.01

kvquant_group_size: int = 32

kvquant_lloyd_iters: int = 8

kvquant_refit_interval: int = 0

kvquant_n_sink: int = 1

palu_rank: int | None = None

palu_energy_threshold: float = 0.9

palu_n_head_groups: int = 4

palu_hi_bit: int = 4

palu_lo_bit: int = 2

palu_hi_fraction: float = 0.25

palu_group_size: int = 32

palu_quantize_values: bool = True

cachegen_bits: int = 4

cachegen_group_size: int = 32

cachegen_use_delta: bool = True

cachegen_per_channel: bool = True

cachegen_layer_groups: int = 3

cachegen_resolved_bits: int | None = None

minicache_start_frac: float = 0.5

minicache_group_size: int = 2

minicache_retention_threshold: float = 0.9

minicache_slerp_t: float = 0.5

minicache_max_ctx: int = 8192

gear_bits: int = 2

gear_rank: int | None = None

gear_energy_threshold: float = 0.9

gear_sparse_fraction: float = 0.01

gear_group_size: int = 32

gear_quantize_values: bool = True

zipcache_hi_bits: int = 4

zipcache_lo_bits: int = 2

zipcache_hi_fraction: float = 0.2

zipcache_group_size: int = 32

zipcache_quantize_values: bool = True

snap_dtype: str = 'auto'

snap_batched_scoring: bool = True

snap_backend: str = 'auto'

snap_budget: int = 512

snap_obs_window: int = 32

snap_n_sink: int = 4

stream_n_sink: int = 4

stream_window_size: int = 512

h2o_budget: int = 512

h2o_n_sink: int = 4

h2o_rope_base: float = 10000.0

h2o_grace: int = 16

h2o_decay: float = 0.98

tova_budget: int = 512

tova_n_sink: int = 4

tova_backend: str = 'auto'

pyramid_budget: int = 512

pyramid_backend: str = 'reference'

pyramid_n_sink: int = 4

pyramid_beta: float = 2.0

pyramid_resolved_budget: int | None = None

squeeze_budget: int = 512

squeeze_n_sink: int = 4

squeeze_strength: float = 1.0

squeeze_resolved_budget: int | None = None

chunkkv_budget: int = 512

chunkkv_chunk_size: int = 8

chunkkv_n_sink: int = 4

chunkkv_score: str = 'attn_mass'

chunkkv_reuse_layers: int = 1

cam_budget: int = 512

cam_n_sink: int = 4

cam_merge: str = 'sim_weighted'

cam_merge_keys: bool = False

cam_merge_gate: bool = True

xkv_group_size: int = 2

xkv_rank: int | None = None

xkv_energy_threshold: float = 0.95

xkv_latent_bits: int = 4

xkv_group_quant_size: int = 32

xkv_max_ctx: int = 8192

nsn_bits: int = 2

nsn_residual_length: int = 64

nsn_codebook_size: int = 256

nsn_subvector_dim: int = 8

nsn_seed: int = 1234

nsn_max_ctx: int = 8192

knorm_budget: int = 512

knorm_n_sink: int = 4

knorm_recent: int = 0

knorm_keep: str = 'low'

skvq_bits_key: int = 2

skvq_bits_value: int = 2

skvq_group_size: int = 32

skvq_window: int = 128

skvq_n_sink: int = 5

skvq_reorder: bool = True

skvq_clip_search: bool = True

skvq_clip_alpha: float = 1.0

skvq_max_ctx: int | None = None

qfilters_budget: int = 512

qfilters_n_sink: int = 4

qfilters_recent: int = 0

qfilters_calib_tokens: int = 128

qfilters_sign: int = 1

qfilters_min_retention: float | None = 0.9

qfilters_on_low_retention: Literal['warn', 'raise', 'ignore'] = 'warn'

keyformer_budget: int = 512

keyformer_n_sink: int = 4

keyformer_recent: int = 0

keyformer_tau: float | None = None

keyformer_tau_init: float = 1.0

keyformer_tau_end: float = 1.0

keyformer_anneal_steps: int = 0

keyformer_rope_base: float = 10000.0

keyformer_seed: int = 0

morphkv_budget: int = 512

morphkv_n_sink: int = 4

morphkv_window: int = 8

kvzip_budget: int = 512

kvzip_n_sink: int = 4

kvzip_probe: str = 'context'

kvtc_bit_budget: int = 512

kvtc_bit_choices: tuple = (0, 1, 2, 3, 4, 6, 8)

kvtc_beta: float = 3.5

curdkv_budget: int = 512

curdkv_n_sink: int = 4

curdkv_rank_cap: int = 16

curdkv_rope_base: float = 10000.0

nestedkv_budget: int = 512

nestedkv_n_sink: int = 4

nestedkv_window: int = 64

nestedkv_beta: float = 3.0

nestedkv_tau: float = 0.6

nestedkv_kappa: float = 10.0

nestedkv_safeguard_alpha: float = 0.2

amc_k_high: float = 0.2

amc_k_mid: float = 0.3

amc_use_query_saliency: bool = False

amc_query_alpha: float = 0.5

amc_adaptive_thresholds: bool = False

amc_threshold_window: int = 64

amc_gamma: float = 0.1

amc_calib_variance: float | None = None

amc_group_size: int = 32

a2ats_codebook_bits: int = 8

a2ats_sub_dim: int = 8

a2ats_window: int = 128

a2ats_b: int = 2048

a2ats_use_query_aware: bool = True

a2ats_beta: float = 0.5

a2ats_retrieval_fraction: float = 0.2

a2ats_rope_base: float = 10000.0

a2ats_rope_scaling: Any = None

a2ats_query_h: Any = None

a2ats_codebook: Any = None

anchorkv_theta: float = 0.05

anchorkv_window: int = 32

anchorkv_rho: float = 0.7

anchorkv_anchor_frac: float = 0.0078125

anchorkv_residual_bits: int = 2

anchorkv_seed: int = 42

rocketkv_compression_ratio: float = 8.0

rocketkv_page_size: int | None = None

rocketkv_head_topk1: int | None = None

rocketkv_obs_window: int = 32

rocketkv_n_sink: int = 4

age_recent_boundary: int = 128

age_mid_boundary: int = 1024

age_bits_recent: int = 8

age_bits_mid: int = 4

age_bits_old: int = 2

age_group_size: int = 32

n_sink_tokens: int = 5

smooth_factors: Any = None

key_codebook: Any = None

value_codebook: Any = None

spectral_key_d_eff: int = 4

spectral_val_d_eff: int = 50

spectral_apply_qjl: bool = True

spectral_model_name: str = 'model'

use_metal_kernels: bool | None = None

fused_sdpa: bool | None = False

fused_sdpa_max_ctx: int | None = None

fused_sdpa_memory_bound: bool = False

class veloxquant_mlx.KVCacheFactory

Bases: object

Factory for creating KVCache instances from a KVCacheConfig.

static create(config)

Instantiate a KVCache from the given configuration.

Parameters:

config (KVCacheConfig) – KVCacheConfig instance.

Return type:

KVCache | _MLXKVCache

Returns:

Configured KVCache. The concrete type depends on config.method – STANDALONE_METHODS produce VeloxQuant’s own KVCache (core. abstractions), every other method produces an mlx_lm.models. cache.KVCache subclass — see the STANDALONE_METHODS comment above.

class veloxquant_mlx.ArtifactStore

Bases: ABC

DAO interface for loading and saving precomputed quantization artifacts.

abstractmethod load_rotation_matrix(d, seed)

Load a precomputed rotation matrix.

Parameters:

• d (int) – Matrix dimension.

• seed (int) – Random seed used to generate the matrix.

Return type:

Any

Returns:

MLX array of shape (d, d), fp16.

Raises:

ArtifactNotFoundError – If the artifact does not exist.

abstractmethod save_rotation_matrix(Pi, d, seed)

Persist a rotation matrix.

Parameters:

• Pi (Any) – Array of shape (d, d).

• d (int) – Dimension.

• seed (int) – Seed used to generate Pi.

Return type:

None

abstractmethod load_codebook(distribution, b, d)

Load a precomputed codebook.

Parameters:

• distribution (str) – Distribution name (e.g. ‘gaussian’, ‘beta’).

• b (int) – Bit-width.

• d (int) – Vector dimension.

Return type:

Any

Returns:

MLX array of shape (2^b,), fp16.

Raises:

ArtifactNotFoundError – If the artifact does not exist.

abstractmethod save_codebook(cb, distribution, b, d)

Persist a codebook.

Parameters:

• cb (Any) – Codebook centroids array.

• distribution (str) – Distribution name.

• b (int) – Bit-width.

• d (int) – Dimension.

Return type:

None

abstractmethod load_jl_matrix(d, m, seed)

Load a precomputed JL projection matrix.

Parameters:

• d (int) – Input dimension.

• m (int) – Output (sketch) dimension.

• seed (int) – Random seed.

Return type:

Any

Returns:

MLX array of shape (m, d), fp16.

Raises:

ArtifactNotFoundError – If the artifact does not exist.

abstractmethod save_jl_matrix(S, d, m, seed)

Persist a JL projection matrix.

Parameters:

• S (Any) – Array of shape (m, d).

• d (int) – Input dimension.

• m (int) – Output dimension.

• seed (int) – Seed.

Return type:

None

abstractmethod exists(artifact_type, **kwargs)

Check whether a specific artifact exists in the store.

Parameters:

• artifact_type (str) – One of ‘rotation’, ‘codebook’, ‘jl’.

• **kwargs (Any) – Identifying parameters (d, seed, b, distribution, m).

Return type:

bool

Returns:

True if the artifact is present and loadable.

class veloxquant_mlx.KVCache

Bases: ABC

Abstract base class for KV cache implementations.

abstractmethod append_key(k)

Append a new key vector to the cache.

Parameters:

k (Any) – Key vector, shape (d,), fp16.

Return type:

None

abstractmethod append_value(v)

Append a new value vector to the cache.

Parameters:

v (Any) – Value vector, shape (d,), fp16.

Return type:

None

abstractmethod attend(q)

Compute attention-weighted value for a query.

Parameters:

q (Any) – Query vector, shape (d,), fp16.

Return type:

Any

Returns:

Attention output, shape (d,), fp16.

abstractmethod memory_bytes()

Return current memory footprint of the cache in bytes.

Return type:

int

reset()

Clear all stored tokens, returning the cache to its empty state.

Not abstract: only concrete storage-owning caches (TurboQuant, PolarQuant, QJL, Spectral, and decorators that delegate to one — SlidingWindowKVCache, PooledKVCache) implement this. A decorator with an unrelated, already-established meaning for “reset” (e.g. KVCacheProfiler.reset(), which clears only its own accumulated profiling stats and deliberately leaves the wrapped cache untouched) is not required to satisfy this contract, so this default raises rather than being abstract.

Implementers must clear only token storage (ring buffers / lists / counters) and leave any quantizer, calibration, or configuration state untouched — callers such as SlidingWindowKVCache rely on reset() to rebuild an emptied cache without re-running (or losing) prior calibration.

Return type:

None

append(k, v)

Append a key-value pair in one call.

Parameters:

• k (Any) – Key vector, shape (d,).

• v (Any) – Value vector, shape (d,).

Return type:

None

class veloxquant_mlx.Quantizer

Bases: ABC

Abstract base class for all vector quantizers.

All public-facing quantizer objects must satisfy this interface.

abstractmethod encode(x)

Encode a batch of vectors into a compact representation.

Parameters:

x (Any) – Input array of shape (batch, d), fp16.

Return type:

EncodedVector

Returns:

EncodedVector containing the compressed representation.

abstractmethod decode(ev)

Reconstruct approximate vectors from an EncodedVector.

Parameters:

ev (EncodedVector) – Encoded representation produced by encode().

Return type:

Any

Returns:

Reconstructed array of shape (batch, d), fp16.

abstractmethod estimate_inner_product(q, ev)

Estimate inner products between a query and encoded keys.

Parameters:

• q (Any) – Query vector, shape (d,) or (1, d), fp16.

• ev (EncodedVector) – Encoded key cache, batch_size == n_keys.

Return type:

Any

Returns:

Estimated inner products, shape (batch_size,), fp16.

class veloxquant_mlx.QuantizationObserver

Bases: ABC

Observer for pipeline events (timing, memory, distortion).

abstractmethod on_event(event)

Handle a quantization pipeline event.

Parameters:

event (Any) – QuantizationEvent dataclass instance.

Return type:

None

class veloxquant_mlx.EncodedVector(quantizer_type, batch_size, dim, indices=None, norm=None, signs=None, residual_norm=None, angles=None, final_radius=None, outlier_idx=None, outlier_encoded=None, inlier_encoded=None)

Bases: object

Typed output of Quantizer.encode().

Different quantizer types populate different subsets of fields.

Variables:

• quantizer_type – Registry key of the producing quantizer.

• batch_size – Number of vectors encoded.

• dim – Original vector dimensionality.

• indices – Scalar codebook indices, shape (batch, d) uint8.

• norm – Per-vector L2 norm, shape (batch,) fp16.

• signs – QJL sign bits, shape (batch, m) int8.

• residual_norm – QJL residual norm, shape (batch,) fp16.

• angles – PolarQuant level angles, list of (batch, d/2^ℓ) arrays.

• final_radius – PolarQuant scalar radius, shape (batch,) fp16.

• outlier_idx – Outlier channel indices for CompositeQuantizer.

• outlier_encoded – Nested EncodedVector for outlier channels.

• inlier_encoded – Nested EncodedVector for inlier channels.

Parameters:

• quantizer_type (str)

• batch_size (int)

• dim (int)

• indices (Any | None)

• norm (Any | None)

• signs (Any | None)

• residual_norm (Any | None)

• angles (list[Any] | None)

• final_radius (Any | None)

• outlier_idx (ndarray | None)

• outlier_encoded (EncodedVector | None)

• inlier_encoded (EncodedVector | None)

quantizer_type: str

batch_size: int

dim: int

indices: Any | None = None

norm: Any | None = None

signs: Any | None = None

residual_norm: Any | None = None

angles: list[Any] | None = None

final_radius: Any | None = None

outlier_idx: ndarray | None = None

outlier_encoded: EncodedVector | None = None

inlier_encoded: EncodedVector | None = None

memory_bytes()

Compute exact memory footprint of this encoded representation.

Return type:

int

Returns:

Total bytes occupied by all stored arrays.

class veloxquant_mlx.QuantizationContext(x_original, mode, x_current, norm=None, rotated=None, indices=None, signs=None, residual_norm=None, angles=None, final_radius=None, outlier_idx=None, packed_bits=None, metadata=<factory>)

Bases: object

Mutable payload flowing through a QuantizationHandler chain.

Variables:

• x_original – Original input vectors, shape (batch, d).

• mode – Whether the chain is encoding or decoding.

• x_current – Working copy mutated by each handler.

• norm – L2 norm stored by NormalizationHandler, shape (batch,).

• rotated – Vector after RotationHandler, shape (batch, d).

• indices – Codebook indices, shape (batch, d) uint8.

• signs – QJL sign bits, shape (batch, m) int8.

• residual_norm – Residual L2 norm for QJL stage, shape (batch,).

• angles – Per-level polar angles from PolarTransformHandler.

• final_radius – Scalar radius after all polar levels, shape (batch,).

• outlier_idx – Channel positions of outlier channels.

• packed_bits – Bit-packed index array from BitPackingHandler.

• metadata – Arbitrary stage-specific metadata.

Parameters:

• x_original (Any)

• mode (Literal['encode', 'decode'])

• x_current (Any)

• norm (Any | None)

• rotated (Any | None)

• indices (Any | None)

• signs (Any | None)

• residual_norm (Any | None)

• angles (list[Any] | None)

• final_radius (Any | None)

• outlier_idx (ndarray | None)

• packed_bits (ndarray | None)

• metadata (dict[str, Any])

x_original: Any

mode: Literal['encode', 'decode']

x_current: Any

norm: Any | None = None

rotated: Any | None = None

indices: Any | None = None

signs: Any | None = None

residual_norm: Any | None = None

angles: list[Any] | None = None

final_radius: Any | None = None

outlier_idx: ndarray | None = None

packed_bits: ndarray | None = None

metadata: dict[str, Any]

class veloxquant_mlx.TransformResult(angles, final_radius, n_levels)

Bases: object

Output of RecursivePolarTransform.forward().

Variables:

• angles – List of angle arrays, one per polar level. angles[0] has shape (batch, d/2), angles[ℓ] has shape (batch, d/2^(ℓ+1)).

• final_radius – Scalar radius at the end of recursion, shape (batch,).

• n_levels – Number of polar recursion levels applied.

Parameters:

• angles (list[Any])

• final_radius (Any)

• n_levels (int)

angles: list[Any]

final_radius: Any

n_levels: int

exception veloxquant_mlx.ArtifactNotFoundError

Bases: FileNotFoundError

Raised when a required precomputed artifact is missing from the store.

exception veloxquant_mlx.BlockPoolExhaustedError

Bases: RuntimeError

Raised when a BlockPoolAllocator has no free blocks left to allocate.

exception veloxquant_mlx.CodebookDimensionMismatch

Bases: ValueError

Raised when a codebook’s shape does not match the expected dimension.

exception veloxquant_mlx.CyclicPipelineError

Bases: RuntimeError

Raised when a QuantizationGraph contains a cycle.

exception veloxquant_mlx.QuantizerConfigError

Bases: ValueError

Raised when a quantizer or KV cache is misconfigured.

class veloxquant_mlx.BlockPoolAllocator(config=None)

Bases: object

Fixed-size block allocator for KV-cache storage.

Pre-allocates n_blocks fixed-size blocks up front and hands them out on request, avoiding per-token malloc/free during generation. Freed blocks return to a free list and are reused by later allocations (LIFO — a just-freed block tends to be cache-hot), so steady-state generation performs zero new-memory allocation once the pool is warm.

Keys and values can be tracked on independent free lists (PoolConfig.separate_kv=True, the default) so K-heavy and V-heavy workloads (e.g. asymmetric bit-widths) don’t compete for the same blocks, or share a single free list for tighter packing when K/V pressure is uneven across requests.

Thread-safe: every mutating and read method acquires an internal lock, so one pool can be shared across threads/concurrent request handlers without external synchronization. The lock is a plain (non-reentrant) threading.Lock — do not call back into the pool from inside a callback invoked while holding it.

This class only manages block bookkeeping (which block ids are free, who owns which, how full each is, fragmentation/latency stats). It does not itself own array storage — pair it with a backing store (e.g. veloxquant_mlx.memory.mlx_storage.MLXBlockStorage) that allocates the actual mx.array buffers indexed by block id.

Parameters:

config (PoolConfig | None) – Pool sizing/behavior configuration.

Example:

pool = BlockPoolAllocator(PoolConfig(block_size=16, n_blocks=256))
blocks = pool.allocate(stream="k", n_tokens=40, owner=request_id, format="int2")
...
pool.free_all(owner=request_id)

register_owner(owner)

Mark owner as active, guarding against accidental id reuse.

allocate() calls this automatically for any owner not yet seen, but a second allocate() for an owner that is already active is treated as that same request continuing to grow, not a collision (see allocate()’s docstring) — it can’t otherwise tell the two cases apart. Call register_owner() explicitly yourself (once, before any allocate() calls for this owner) if you want a genuine collision — e.g. a request-id counter that wrapped around, or two callers racing on the same id — to raise immediately, including on a second register_owner() call for a still-active id.

Raises:

OwnerAlreadyActiveError – If owner already has blocks checked out or was registered and not yet released.

Return type:

None

Parameters:

owner (int)

release_owner(owner)

Clear owner from the active-owner set without freeing blocks.

Rarely needed directly — free_all() calls this once an owner’s last block is freed. Useful if you registered an owner via register_owner() but never actually allocated anything for it.

Return type:

None

Parameters:

owner (int)

allocate(stream, n_tokens, owner, format='fp16')

Allocate enough blocks to hold n_tokens slots.

Parameters:

• stream (str) – “k” or “v” (ignored, treated as “kv”, when the pool was built with separate_kv=False).

• n_tokens (int) – Number of token slots needed.

• owner (int) – Opaque id (e.g. request id) the blocks are checked out to. Repeated allocate() calls for an owner that is already active on this pool are treated as that same request growing (the normal case — see PoolBackedKVCache) and never raise. To catch a different caller accidentally reusing an id that is still checked out, call register_owner() yourself before the first allocate() — a second register_owner() for an id that is still active raises OwnerAlreadyActiveError.

• format (str) – Compression-format tag to record on each block.

Return type:

list[Block]

Returns:

List of newly allocated Block instances, in order.

Raises:

• BlockPoolExhaustedError – If fewer than the required number of free blocks remain and the pool cannot grow enough to cover the gap (see PoolConfig.grow_on_exhaustion/max_blocks). No blocks are allocated on failure (all-or-nothing).

• ValueError – If n_tokens <= 0 or stream is invalid.

free(blocks)

Return blocks to the free list for their stream.

Parameters:

blocks (list[Block]) – Blocks previously returned by allocate().

Return type:

None

free_all(owner)

Free every block currently checked out to owner.

Parameters:

owner (int) – Opaque id passed to a prior allocate() call.

Return type:

None

Unlike free(), which frees an arbitrary caller-supplied subset and must locate+remove each block id from the owner’s list, this already has the owner’s complete, correct block-id list – so it pops that list once (O(1)) instead of doing N membership checks + N list.remove() calls (VeloxQuant-MLX#510; was 13.4ms at 256K tokens / 16K blocks, confirmed superlinear: 2x blocks -> 2.0-3.4x time).

blocks_for(owner)

Return the blocks currently checked out to owner, in allocation order.

Return type:

list[Block]

Parameters:

owner (int)

n_free(stream='k')

Number of free blocks remaining for stream.

Return type:

int

Parameters:

stream (str)

shrink(stream, target_free)

Permanently retire fully-free blocks from stream down to target_free.

Use this to give memory back when a pool was sized for peak load that turned out much higher than steady-state usage. Only blocks currently on the free list are eligible — in-use blocks are never touched, so this never breaks a live request. Retired blocks are removed from the pool’s bookkeeping entirely (not just marked free/unfree), so any external backing storage indexed by block id (e.g. MLXBlockStorage) may drop the corresponding buffer.

Parameters:

• stream (str) – “k” or “v” (ignored, treated as “kv”, when the pool was built with separate_kv=False).

• target_free (int) – Desired number of free blocks to retain for this stream after shrinking. If the stream already has this many or fewer free blocks, this is a no-op.

Return type:

int

Returns:

Number of blocks actually retired.

Raises:

ValueError – If target_free < 0.

mark_used(block, n_used)

Update block.n_used and keep fragmentation stats consistent.

Callers that track a block’s fill level themselves (e.g. PooledKVCache, which appends token-by-token to an already-checked-out block) must go through this instead of writing block.n_used directly – AllocationStats._fragmented_count is maintained incrementally (not recomputed by a full scan every allocate()/free()/shrink() call, which would be O(n_blocks) on every single-block growth step in the decode hot path), so it only stays correct if every write to a block’s fill level is observed here.

Parameters:

• block (Block) – A block currently checked out from this pool (owner is not None). Blocks that are free are never fragmented by definition, so calling this on one is a no-op for stats (n_used is still updated).

• n_used (int)

Return type:

None

class veloxquant_mlx.PoolConfig(block_size=16, n_blocks=1024, separate_kv=True, grow_on_exhaustion=False, max_blocks=None)

Bases: object

Configuration for a BlockPoolAllocator.

Variables:

• block_size – Number of token slots stored per block.

• n_blocks – Total number of blocks the pool manages initially.

• separate_kv – If True, keys and values are allocated from independent free lists (K and V blocks are never interchanged). If False, a single free list backs both keys and values, which packs tighter when K and V pressure is uneven across requests.

• grow_on_exhaustion – If True, an allocate() call that would otherwise raise BlockPoolExhaustedError instead grows the exhausted stream’s block count just enough to satisfy the request (see max_blocks). If False (default), exhaustion always raises.

• max_blocks – Upper bound on total blocks across all streams once grown. None means unbounded growth. Ignored when grow_on_exhaustion is False. Growth that would exceed this cap still raises BlockPoolExhaustedError.

Parameters:

• block_size (int)

• n_blocks (int)

• separate_kv (bool)

• grow_on_exhaustion (bool)

• max_blocks (int | None)

block_size: int = 16

n_blocks: int = 1024

separate_kv: bool = True

grow_on_exhaustion: bool = False

max_blocks: int | None = None

class veloxquant_mlx.PooledKVCache(inner, pool, owner=-1, format='fp16')

Bases: KVCache

Wraps any VeloxQuant KVCache with block-pool-backed memory accounting.

The wrapped cache still owns and drives its own storage arrays (its compression format may be anything registered with KVCacheFactory); this wrapper checks out and returns fixed-size blocks from a shared BlockPoolAllocator in lock-step with token appends, so a multi-request server can track allocation counts, reuse, and fragmentation across every active cache from one pool instead of each cache doing its own ad hoc growth.

Block granularity is pool.config.block_size tokens: a new block is requested from the pool only when the previous one fills up, and every block owned by this cache is returned to the pool in one call when the request completes (release()). No block is ever allocated or freed on a per-token basis, matching the “avoid malloc/free during generation” goal in issue #249.

Parameters:

• inner (KVCache) – The underlying KVCache that performs actual compression.

• pool (BlockPoolAllocator) – Shared BlockPoolAllocator to draw block accounting from.

• owner (int) – Opaque id identifying this cache’s request/sequence. Two PooledKVCache instances must use different owners to avoid each other’s release() reclaiming the wrong blocks.

• format (str) – Compression-format tag recorded on each block this cache checks out (purely informational — passed through to pool.allocate(format=...)).

append_key(k)

Check out a K block from the pool if needed, then append to the inner cache.

Parameters:

k (Any) – Key vector, shape (d,), fp16.

Return type:

None

append_value(v)

Check out a V block from the pool if needed, then append to the inner cache.

Parameters:

v (Any) – Value vector, shape (d,), fp16.

Return type:

None

attend(q)

Delegate attention computation to the inner cache.

Parameters:

q (Any) – Query vector, shape (d,), fp16.

Return type:

Any

Returns:

Attention output, shape (d,), fp16.

memory_bytes()

Return memory usage of the inner (compressed) cache.

Return type:

int

release()

Return every block this cache holds to the pool.

Call this once the request finishes (or is evicted) so its blocks become available for reuse by later requests.

Return type:

None

n_blocks_held()

Total K + V blocks currently checked out to this cache.

Return type:

int

reset()

Return all held blocks to the pool and clear the inner cache.

Return type:

None

class veloxquant_mlx.QuantizerFactory

Bases: object

Factory for creating Quantizer instances via the QuantizerRegistry.

All quantizer construction should go through this factory.

Example:

q = QuantizerFactory.create("qjl", d=128, m=128, seed=42)
q = QuantizerFactory.create("turboquant_mse", d=128, b=2, seed=42)
q = QuantizerFactory.create("turboquant_prod", d=128, b=3, m=128, seed=42)
q = QuantizerFactory.create("polar", d=128, b=2, seed=42)

static create(method, d, b=2, m=None, seed=42, store=None, **kwargs)

Instantiate a Quantizer by method name.

Parameters:

• method (Literal['qjl', 'turboquant_mse', 'turboquant_prod', 'polar']) – Algorithm name registered in QuantizerRegistry.

• d (int) – Vector dimension.

• b (int) – Bit-width (default 2).

• m (int | None) – JL projection dimension (default = d for QJL methods).

• seed (int) – Random seed.

• store (ArtifactStore | None) – ArtifactStore for loading precomputed artifacts. If None, artifacts are generated on-the-fly.

• **kwargs (Any) – Additional keyword arguments forwarded to the quantizer.

Return type:

Quantizer

Returns:

Configured Quantizer instance.

Raises:

• QuantizerConfigError – If parameters are invalid.

• KeyError – If method is not registered.

veloxquant_mlx.allocate_bits_ratequant(sensitivities, target_avg_bits, beta=3.5, bit_choices=(1, 2, 3))

Allocate per-layer bit-widths via RateQuant Theorem 2.

Continuous solution (paper, eq. 2):

b_i = b̄ + (ln w_i − ln_w_bar) / ln(β)

rounded to the nearest integer in bit_choices, then re-balanced via greedy +1/−1 adjustments so the integer total exactly matches round(target_avg_bits * N).

Parameters:

• sensitivities – Iterable of per-layer sensitivity weights w_i > 0, typically from calibrate_layer_sensitivities().

• target_avg_bits (float) – Desired mean bits/dim across layers. May be fractional (e.g. 1.5) — the integer allocation will straddle it.

• beta (float) – Distortion-rate decay constant for the underlying quantizer. Paper-reported values: 3.5 for TurboQuant, 5.0 for KIVI/QuaRot. Mismatched β can invert allocation ordering — see paper Section 4.3.

• bit_choices (tuple) – Allowed integer bit-widths. RVQ supports {1, 2, 3, …}.

Return type:

list[int]

Returns:

list[int] of length len(sensitivities). Pass directly as KVCacheConfig.bit_width_inlier and consume via KVCacheBuilder.for_model().

veloxquant_mlx.calibrate_layer_sensitivities(model, tokenizer, prompts=None, seq_len=256, verbose=False)

Run a calibration forward pass and return per-layer sensitivity.

Parameters:

• model – Loaded mlx_lm model (e.g. from mlx_lm.load()).

• tokenizer – Matching tokenizer.

• prompts (list | None) – Optional list of calibration strings. Defaults to 8 general-domain prompts spanning history, science, and CS.

• seq_len (int) – Max tokens per prompt (truncated if longer).

• verbose (bool) – Whether to print per-sequence progress.

Return type:

list[float]

Returns:

list[float] of length n_attention_layers, each strictly > 0. Higher values indicate layers whose key cache is more error-prone at fixed bit-width and should get more bits.

veloxquant_mlx.fit_distortion_curve(head_dim, bit_choices=(1, 2, 3), seed=0, n_samples=64)

Fit D(b) = α·β^(-b) on synthetic unit-norm Gaussian keys.

Return type:

tuple[float, float]

Returns:

Tuple (alpha, beta). For TurboQuantRVQ at d=128 the paper reports β ≈ 3.5; you will get a similar value from this function. For production use, just pass beta=3.5 to allocate_bits_ratequant() and skip the fit.

Parameters:

• head_dim (int)

• bit_choices (tuple)

• seed (int)

• n_samples (int)

class veloxquant_mlx.VecInferKVCache(config)

Bases: KVCache

KV cache implementing VecInfer’s dual-transform product VQ.

Parameters:

config (Any) – KVCacheConfig with VecInfer-specific fields populated by KVCacheFactory.create. Required fields: head_dim, key_codebook_bits, value_codebook_bits, key_sub_dim, value_sub_dim. Optional: smooth_factors (numpy or mx array; identity if absent), key_codebook, value_codebook (random init if absent — for tests only).

Notes

Storage layout deliberately delegates concatenation to mlx_lm’s base _MLXKVCache. We quantize + dequantize on the way in so that subsequent SDPA calls see a standard fp16 key tensor.

Never exposes .bits — mlx_lm’s SDPA checks hasattr(cache, "bits") to route to a different kernel path. We expose .assigned_avg_bits instead.

update_and_fetch(keys, values)

Apply smooth-scale + Hadamard rotation to keys, product-VQ-quantize K/V, and dequantize for downstream SDPA (or stash indices only when memory-bound fused mode is active).

property state

parent fp16 (keys, values). Fused mode: raw (key_indices, value_indices) ring buffers, since no fp16 K_hat/V_hat is materialized.

Type:

Standard mode

empty()

Whether the cache holds no tokens yet (checks the index ring buffer in fused mode, the parent buffer otherwise).

Return type:

bool

trim(n)

Drop the last n tokens, keeping every parallel buffer in step.

The inherited trim only lowers offset. In fused mode the index ring buffers are advanced by _stored_S_kv instead, so left alone they keep the trimmed tokens: the next token is written n slots past offset and fused_sdpa attends over the stale tail. The byte accounting is scaled back with the token count (an estimate: it is tracked per call, not per token).

Return type:

int

Parameters:

n (int)

size()

Number of tokens currently stored.

Return type:

int

property nbytes: int

the parent fp16 K_hat/V_hat buffers plus, in fused mode, the pre-allocated uint32 index buffers (stash mode holds both, so it uses more memory than plain fp16).

Type:

Bytes actually held

fused_sdpa(q, scale, *, causal=True, sliding_window=0)

Compute attention directly from compressed K/V indices.

Parameters:

• q (array) – [B, H_q, S_q, D] uncompressed queries — not yet smooth or Hadamard transformed. This method applies the dual transform internally so callers (mlx_lm SDPA dispatcher) pass exactly the same tensor they’d pass to standard SDPA.

• scale (float) – Attention scale (typically 1/sqrt(D)).

• causal (bool) – Apply causal mask.

• sliding_window (int) – If > 0, sliding-window attention with the given window width.

Return type:

array

Returns:

[B, H_q, S_q, D] attention output in the dtype of q.

property compressed_key_bytes: int

see nbytes.

Type:

Estimated packed bytes for the compressed key cache (product-VQ codes at bits each, all heads/batches; excludes codebook). Not what is held in memory

property fp16_key_bytes: int

Hypothetical fp16 key cost if nothing were compressed.

property compressed_value_bytes: int

see nbytes.

Type:

Estimated packed bytes for the compressed value cache (product-VQ codes at bits each, all heads/batches; excludes codebook). Not what is held in memory

property fp16_value_bytes: int

Hypothetical fp16 value cost if nothing were compressed.

property codebook_bytes: int

Static codebook overhead in bytes (fp16 storage).

property assigned_avg_bits: float

Effective bits/element averaged over keys and values.

Excludes codebook overhead (amortized across many tokens); for an end-to-end byte ratio use compressed_*_bytes / fp16_*_bytes.

property merge

veloxquant_mlx.apply_dual_transform_keys(K, smooth, H)

Apply K_tilde = (K / lambda) @ H (smooth then Hadamard).

Parameters:

• K (array) – Keys, shape [..., head_dim]. Per-head smoothing assumes the second-to-last axis indexes heads when smooth is 2D.

• smooth (array) – Either [head_dim] (broadcast across heads) or [n_heads, head_dim] (per-head); see calibrate_smooth_factors().

• H (array) – Walsh-Hadamard matrix [head_dim, head_dim].

Return type:

array

Returns:

Transformed keys, same shape as input.

veloxquant_mlx.apply_dual_transform_queries(q, smooth, H)

Apply q_tilde = (q * lambda) @ H so q_tilde @ K_tilde.T == q @ K.T.

Parameters:

• q (array) – Queries, shape [..., head_dim]. Under GQA, the head axis (-3) may have H_q = H_kv * n_rep heads, a multiple of smooth’s per-KV-head row count — each KV head’s factor is repeated across its query group (see :issue:`616`), not averaged, so the dual transform still cancels against apply_dual_transform_keys()’s per-KV-head factors.

• smooth (array) – Same convention as apply_dual_transform_keys().

• H (array) – Walsh-Hadamard matrix [head_dim, head_dim].

Return type:

array

Returns:

Transformed queries, same shape as input.

veloxquant_mlx.calibrate_smooth_factors(keys_calib, eps=0.0001)

Compute per-(head, channel) smooth scaling factor from a key sample.

Parameters:

• keys_calib (array) – Calibration keys with shape [n_tokens, n_heads, head_dim] or [n_tokens, head_dim] (single-head).

• eps (float) – Floor for the per-channel max to avoid divide-by-zero.

Return type:

array

Returns:

Smooth factors shaped to match the input head layout ([n_heads, head_dim] or [head_dim]). Formula:

lambda_i = sqrt(max_t |K[t, ..., i]|)

veloxquant_mlx.train_codebook(x, n_centroids, max_iter=30, seed=42)

Train a VQ codebook on flat sub-vector samples.

Parameters:

• x (array) – Flat [n_samples, sub_dim] MLX array of training sub-vectors.

• n_centroids (int) – Codebook size 2**b.

• max_iter (int) – K-means iteration cap.

• seed (int) – RNG seed.

Return type:

array

Returns:

Codebook of shape [n_centroids, sub_dim] as an MLX float32 array.

veloxquant_mlx.walsh_hadamard_matrix(d, dtype=mlx.core.float32)

Construct an orthonormal Walsh-Hadamard matrix.

Recursive form from VecInfer Eq. 5:

H_1 = [[1]]
H_{2k} = (1/sqrt(2)) * [[H_k, H_k], [H_k, -H_k]]

Parameters:

• d (int) – Output dimension; must be a power of 2.

• dtype – MLX dtype for the returned matrix.

Return type:

array

Returns:

Matrix of shape [d, d] with H @ H.T == I.

class veloxquant_mlx.KeyNormObserver

Bases: QuantizationObserver

Accumulates per-token key L2 norm² for later RateQuant allocation.

Usage:

obs = KeyNormObserver()
config.observers.append(obs)
# ... run calibration prompts ...
report = obs.report()
# report.heterogeneity_ratio indicates RateQuant benefit

on_event(event)

Accumulate key_l2_norm_sq from the event’s metadata, if present.

Return type:

None

Parameters:

event (QuantizationEvent)

report()

Build a KeyNormReport from the norms accumulated so far.

Return type:

KeyNormReport

reset()

Clear all accumulated norm statistics.

Return type:

None

class veloxquant_mlx.KeyNormReport(n_tokens, mean_norm_sq, min_norm_sq, max_norm_sq)

Bases: object

Summary statistics produced by KeyNormObserver.

Parameters:

• n_tokens (int)

• mean_norm_sq (float)

• min_norm_sq (float)

• max_norm_sq (float)

n_tokens: int

mean_norm_sq: float

min_norm_sq: float

max_norm_sq: float

property heterogeneity_ratio: float

max / min — predictor of when mixed-precision allocation helps.

Per RateQuant Theorem 3, when this ratio (geometric variant of AM/GM) is far above 1, per-layer bit allocation produces measurable gains over uniform quantization. Below ~2.0, savings are marginal.

class veloxquant_mlx.KVCacheProfiler(cache, head_dim=None, layer_id=0)

Bases: KVCache

Wraps a KVCache instance, recording per-call timing and memory.

Transparent decorator: implements the same append_key/append_value/attend/ memory_bytes interface as the wrapped cache, so it can be substituted anywhere a KVCache is expected (including inside a list built by KVCacheBuilder.for_model()).

Parameters:

• cache (KVCache) – The KVCache instance to profile.

• head_dim (int | None) – Head dimension, used to compute the fp16 compression baseline (2 bytes/element). Defaults to reading cache._d if present, else 0 (disables compression_ratio).

• layer_id (Any) – Optional label for this layer, used in reports.

Example:

cache = KVCacheFactory.create(config)
profiled = KVCacheProfiler(cache, head_dim=config.head_dim, layer_id=0)
profiled.append_key(k)
profiled.append_value(v)
out = profiled.attend(q)
print(format_profile_table([profiled.profile()]))

append_key(k)

Append a key to the wrapped cache, timing it as a quantize call.

Return type:

None

Parameters:

k (Any)

append_value(v)

Append a value to the wrapped cache, timing it as a write call.

Return type:

None

Parameters:

v (Any)

attend(q)

Run attention on the wrapped cache, timing it as a dequantize call.

Return type:

Any

Parameters:

q (Any)

memory_bytes()

Return the wrapped cache’s current memory_bytes() reading.

Return type:

int

profile()

Return the accumulated LayerProfile for this wrapped cache.

Reads the wrapped cache’s memory_bytes() here rather than after every append_key/append_value call: these caches are append-only (memory is monotonically non-decreasing, no eviction), so the latest reading is already the peak, and there is no need to pay for a recompute on every token.

Return type:

LayerProfile

reset()

Clear all accumulated profiling stats (does not affect the wrapped cache).

Return type:

None

class veloxquant_mlx.MLXCacheProfiler(cache, layer_id=0)

Bases: object

Wraps an mlx_lm.models.cache.KVCache-style cache’s update_and_fetch.

Servable methods (the ones cli/serve.py and cli/profile.py drive through real mlx_lm.generate() runs) implement update_and_fetch(keys, values) -> (keys, values), a single fused call that quantizes, stores, and immediately dequantizes for the caller’s attention step — there is no separate append_key/append_value/attend triad to hook into, unlike the standalone interface KVCacheProfiler wraps. So this class times update_and_fetch as one unit and records it as quantize_ms_total with is_fused=True; dequantize_ms_total and write_ms_total stay 0 because there is nothing else to attribute them to. Memory comes from the cache’s own nbytes property (the same true packed-byte accounting cli/serve.py’s accounting-only warning already describes — see ACCOUNTING_NOTE in cli/profile.py).

Substituted directly into the list KVCacheBuilder.for_model() returns and passed to mlx_lm.generate(..., prompt_cache=profilers).

Parameters:

• cache (Any)

• layer_id (Any)

update_and_fetch(keys, values)

Forward to the wrapped cache’s update_and_fetch, timing it as one fused call.

Return type:

Any

Parameters:

• keys (Any)

• values (Any)

profile()

Return the accumulated LayerProfile for this wrapped cache.

Reads the wrapped cache’s nbytes here rather than after every update_and_fetch call: nbytes is monotonically non-decreasing across calls, so the latest reading is already the peak, and there is no need to pay for a recompute on every token.

Return type:

LayerProfile

class veloxquant_mlx.LayerProfile(layer_id, n_quantize_calls=0, n_dequantize_calls=0, quantize_ms_total=0.0, dequantize_ms_total=0.0, write_ms_total=0.0, peak_memory_bytes=0, tokens_written=0, fp16_baseline_bytes=0, is_fused=False)

Bases: object

Aggregated profiling stats for a single KV-cache layer.

Variables:

• layer_id – Index or label identifying the layer.

• n_quantize_calls – Number of append_key calls (quantization events).

• n_dequantize_calls – Number of attend calls (dequantization events).

• quantize_ms_total – Cumulative append_key wall time, milliseconds. For an MLXCacheProfiler (fused update_and_fetch interface), this holds the combined quantize+dequantize+write time instead — see is_fused. There is no separate number to split it into.

• dequantize_ms_total – Cumulative attend wall time, milliseconds. Always 0 when is_fused is True.

• write_ms_total – Cumulative append_value wall time, milliseconds. Always 0 when is_fused is True.

• peak_memory_bytes – memory_bytes()/nbytes of the wrapped cache as of the last profile() call. These caches are append-only (monotonically non-decreasing memory, no eviction), so the latest reading is the peak — read lazily at report time rather than after every call.

• tokens_written – Number of append_key (or update_and_fetch) calls, a proxy for tokens stored.

• fp16_baseline_bytes – What the incoming K+V would cost uncompressed (4 * head_dim bytes per token standalone; the actual K+V tensor bytes for MLXCacheProfiler), used to derive compression_ratio.

• is_fused – True when quantize_ms_total is a combined measurement (MLXCacheProfiler) rather than quantize-only (KVCacheProfiler).

Parameters:

• layer_id (Any)

• n_quantize_calls (int)

• n_dequantize_calls (int)

• quantize_ms_total (float)

• dequantize_ms_total (float)

• write_ms_total (float)

• peak_memory_bytes (int)

• tokens_written (int)

• fp16_baseline_bytes (int)

• is_fused (bool)

layer_id: Any

n_quantize_calls: int = 0

n_dequantize_calls: int = 0

quantize_ms_total: float = 0.0

dequantize_ms_total: float = 0.0

write_ms_total: float = 0.0

peak_memory_bytes: int = 0

tokens_written: int = 0

fp16_baseline_bytes: int = 0

is_fused: bool = False

property quantize_ms_mean: float

Average append_key (or update_and_fetch) latency in milliseconds.

property dequantize_ms_mean: float

Average attend latency in milliseconds (always 0 when is_fused is True).

property compression_ratio: float

fp16 baseline bytes divided by actual bytes (>1 means compressed).

class veloxquant_mlx.ProfileReport(layers=<factory>, elapsed_s=0.0)

Bases: object

Profiling results across one or more layers.

Variables:

• layers – Per-layer profiles, in layer order.

• elapsed_s – Total wall time covered by the profiling session.

Parameters:

• layers (list[LayerProfile])

• elapsed_s (float)

layers: list[LayerProfile]

elapsed_s: float = 0.0

property total_bytes_written: int

Sum of peak memory across all profiled layers.

property total_tokens: int

Tokens processed by the model (not summed across layers).

Every layer sees every token, so summing per-layer counts would overstate the total by the layer count (#629); take the max instead.

property tokens_per_sec: float

Total tokens divided by elapsed_s (0.0 if elapsed_s is unset).

property overall_compression_ratio: float

Combined fp16-baseline-bytes over actual bytes across all layers.

veloxquant_mlx.format_profile_table(report)

Render a ProfileReport as a human-readable fixed-width table.

Matches the format proposed in issue #252:

Layer       Quantize   Dequantize   Memory
------------------------------------------------
Layer 0     12.3 µs    8.1 µs       1.2 MB
Layer 1     11.8 µs    7.9 µs       1.2 MB

Parameters:

report (ProfileReport) – A ProfileReport (or a bare list[LayerProfile]).

Return type:

str

Returns:

Multi-line formatted string.

veloxquant_mlx.profile_layers(profilers, elapsed_s=0.0)

Aggregate a list of KVCacheProfiler instances into one ProfileReport.

Parameters:

• profilers (list[KVCacheProfiler | MLXCacheProfiler]) – One profiler per model layer, in layer order.

• elapsed_s (float) – Total wall-clock time of the profiled run, for tokens/sec.

Return type:

ProfileReport

Returns:

A ProfileReport summarizing all layers.

class veloxquant_mlx.AutoConfigResult(config, reason)

Bases: object

The selected configuration plus the reasoning behind it.

Variables:

• config – The resulting KVCacheConfig, ready for KVCacheFactory.create() or KVCacheBuilder.for_model().

• reason – Short human-readable explanation of why this method/bit-width was chosen, for logging or a CLI’s --explain flag.

Parameters:

• config (KVCacheConfig)

• reason (str)

config: KVCacheConfig

reason: str

class veloxquant_mlx.HardwareInfo(total_memory_bytes=None, active_memory_bytes=0)

Bases: object

Describes the machine the KV cache will run on.

Variables:

• total_memory_bytes – Total device memory (unified memory on Apple Silicon). None disables memory-pressure-based selection.

• active_memory_bytes – Memory already in use before this cache is allocated (e.g. model weights, other requests’ caches).

Parameters:

• total_memory_bytes (int | None)

• active_memory_bytes (int)

total_memory_bytes: int | None = None

active_memory_bytes: int = 0

pressure_fraction(additional_bytes)

Fraction of total memory used once additional_bytes is added.

Returns None if total_memory_bytes is unknown.

Return type:

float | None

Parameters:

additional_bytes (int)

class veloxquant_mlx.WorkloadSpec(head_dim=128, seq_len=4096, n_layers=1, batch_size=1, n_kv_heads=1)

Bases: object

Describes the job the KV cache needs to serve.

Variables:

• head_dim – Attention head dimension. Must be a positive power of 2, matching KVCacheConfig’s own constraint.

• seq_len – Expected (or worst-case) sequence length in tokens. Drives the short-context / long-context precision tradeoff.

• n_layers – Number of attention layers whose caches will share this configuration. Used only to size the memory-pressure estimate; defaults to 1 (single-layer estimate).

• batch_size – Number of concurrent sequences. Used only to size the memory-pressure estimate; defaults to 1.

• n_kv_heads – Number of KV heads per layer. Multiplies the fp16 K+V footprint used by the memory-pressure rule, so set it to the model’s real value (e.g. 8 for Llama-3-8B); the default of 1 keeps the older single-head estimate and undercounts by this factor for multi-head models (#631).

Parameters:

• head_dim (int)

• seq_len (int)

• n_layers (int)

• batch_size (int)

• n_kv_heads (int)

head_dim: int = 128

seq_len: int = 4096

n_layers: int = 1

batch_size: int = 1

n_kv_heads: int = 1

fp16_kv_bytes()

Estimated fp16 footprint of the full K+V cache for this workload.

Return type:

int

veloxquant_mlx.detect_hardware_info()

Auto-detect HardwareInfo from the running MLX device.

Falls back to an empty (unknown-memory) HardwareInfo if MLX is unavailable or device introspection fails — callers then get sequence-length-only selection instead of an exception.

Return type:

HardwareInfo

veloxquant_mlx.select_kv_cache_config(workload, hardware=None)

Pick a KVCacheConfig for the given workload and hardware.

Selection rules (in priority order):

1. Memory pressure: if hardware reports the workload’s estimated fp16 footprint would push total usage above MEMORY_PRESSURE_FRACTION of device memory, pick the most aggressive method in the pool (gear, 2-bit) regardless of sequence length.

2. Sequence length: short contexts (< SHORT_CONTEXT_TOKENS) favor higher precision (turboquant_rvq, 4-bit); long contexts (>= LONG_CONTEXT_TOKENS) favor aggressive compression (kvquant, 3-bit with outlier isolation); everything in between uses a balanced default (kivi, 2-bit).

3. Head dimension: if head_dim >= LARGE_HEAD_DIM, the group size is doubled (min(head_dim, 64)) so each group still holds enough elements to amortize per-group scale/zero-point overhead.

Parameters:

• workload (WorkloadSpec) – Description of the job (head_dim, seq_len, n_layers, batch_size, n_kv_heads).

• hardware (HardwareInfo | None) – Description of the target machine. If None, auto-detects via detect_hardware_info().

Return type:

AutoConfigResult

Returns:

AutoConfigResult with the selected KVCacheConfig and a human-readable reason string.

class veloxquant_mlx.AutoOptimizer(options=None)

Bases: object

One-stop automatic KV-cache strategy recommender with caching and live validation.

Preview API: see the module-level warning above — recommendations are an analytical proxy (±20% memory error band, table-estimated hardware bandwidth), not a measured guarantee. Prefer a low probe_top_n or a real benchmark database before production use.

Purpose: Answers “which of the 43 methods should I use?” by profiling hardware/model, filtering incompatible methods, ranking by objective, and validating top choices against the live serve-tier to prevent crashes.

Deliberately stateful: Hardware detection and benchmark database are cached on the instance, so repeated calls don’t re-detect or re-probe. This makes it efficient for interactive CLI use and Jupyter notebooks.

Design: 1. Hardware caching: First detect_hardware() call introspects once 2. Model profiling: Each call profiles the given config (no caching; configs vary) 3. Strategy planning: Filters, estimates memory, scores by objective 4. Serve-tier validation: Probes top N candidates to catch crash-tier methods 5. Fallback: If nothing survives, recommends library default (turboquant_rvq)

Usage:

>>> optimizer = AutoOptimizer()
>>> result = optimizer.recommend_strategy(model_config=hf_config, objective="latency")
>>> print(optimizer.explain(result))

With benchmarks (optional speedup):

>>> from pathlib import Path
>>> optimizer = AutoOptimizer(options=AutoOptimizerOptions(
...     benchmark_db_dir=str(Path.home() / ".cache" / "veloxquant" / "benchmarks")
... ))

Testing (no probing):

>>> optimizer = AutoOptimizer(options=AutoOptimizerOptions(probe_top_n=0))
>>> result = optimizer.recommend_strategy(model_config=...)  # Fast, but may recommend crash-prone methods

Attributes:

options: AutoOptimizerOptions (benchmark_db_dir, probe_top_n)

Parameters:

options (AutoOptimizerOptions | None)

detect_hardware()

Detect (and cache) this machine’s HardwareProfile.

Return type:

HardwareProfile

profile_model(config=None, **overrides)

Profile a model from a HF-style config dict + optional overrides.

Return type:

ModelProfile

Parameters:

• config (dict[str, Any] | None)

• overrides (Any)

estimate_memory(workload, model=None)

Analytical per-method memory estimates for a workload.

Return type:

dict[str, MemoryEstimate]

Parameters:

• workload (WorkloadProfile)

• model (ModelProfile | None)

recommend_strategy(model_config=None, *, model=None, workload=None, objective=None, **kwargs)

Recommend + validate the best KV-cache strategy for a model + workload.

Pipeline: 1. Profile model (from config or prebuilt ModelProfile) 2. Detect hardware (cached after first call) 3. Look up benchmarks in local DB (if enabled) 4. Filter incompatible methods (attention type, memory budget) 5. Score survivors on 4 axes (memory, latency, throughput, quality) 6. Rank by objective weights (memory vs latency vs quality) 7. Probe top N candidates with serve-tier (catch crashes) 8. Return recommendation + ranked alternatives + explanation

Return type:

RecommendationResult

Parameters:

• model_config (dict[str, Any] | None)

• model (ModelProfile | None)

• workload (WorkloadProfile | None)

• objective (str | None)

• kwargs (Any)

Arguments:

model_config: HuggingFace-style config dict for architecture extraction

(num_layers, hidden_size, num_attention_heads, etc.). One of model_config or model must be provided.

model: Prebuilt ModelProfile; skips config parsing if given. workload: WorkloadProfile with context_length, batch_size, objective.

Defaults to balanced 4K context if not given.

objective: Convenience override for workload.objective

(“memory”, “latency”, “quality”, or “balanced”).

**kwargs: Additional PlanningOptions forwarded to the planner:

• memory_budget_bytes: Hard cap on KV footprint

• prefer_no_calibration: Exclude methods needing setup

• require_metal: Exclude CPU-fallback implementations

Returns:

RecommendationResult with: - recommendation: Top choice (ScoredMethod) - ranked: Full ranked list (1–3 alternatives) - fallback_used: True if no viable method survived - candidates: Filtered set (viable + excluded) - model/hardware/workload: The inputs (for explanation)

Examples:

Basic recommendation: >>> result = optimizer.recommend_strategy( … model_config={“num_layers”: 32, “hidden_size”: 4096, “num_attention_heads”: 32} … )

With objective override: >>> result = optimizer.recommend_strategy(model_config=config, objective=”latency”)

With memory budget: >>> result = optimizer.recommend_strategy( … model_config=config, … memory_budget_bytes=8 * 1024**3 # 8 GiB max … )

Get explanation: >>> print(optimizer.explain(result))

explain(result, **kwargs)

Render explain() text.

Return type:

str

Parameters:

• result (RecommendationResult)

• kwargs (Any)

fallback_method()

Library-default method name used when nothing survives filtering.

Return type:

str

class veloxquant_mlx.AutoOptimizerOptions(benchmark_db_dir=None, probe_top_n=3)

Bases: object

Constructor options for AutoOptimizer.

Variables:

• benchmark_db_dir – Path to a BenchmarkDatabase storage directory; enabled only when explicitly given, so a stray dir never changes recommendations behind the user’s back.

• probe_top_n – How many of the ranked candidates are validated against the live serve-tier probe before committing. 0 skips probing (faster, but a crash-prone standalone method could be recommended).

Parameters:

• benchmark_db_dir (str | None)

• probe_top_n (int)

benchmark_db_dir: str | None = None

probe_top_n: int = 3

class veloxquant_mlx.HardwareProfile(chip='unknown', chip_generation=0, total_memory_bytes=None, available_memory_bytes=0, mlx_version=None, macos_version=None, metal_available=True, peak_memory_bandwidth_gbps=None, avg_quantize_latency_ms_per_token=None)

Bases: object

Comprehensive description of the Apple-Silicon environment.

Unlike the minimal HardwareInfo, this is the object the auto-selection pipeline passes around: every field is a fact the planner can cite in an explanation.

Parameters:

• chip (str)

• chip_generation (int)

• total_memory_bytes (int | None)

• available_memory_bytes (int)

• mlx_version (str | None)

• macos_version (str | None)

• metal_available (bool)

• peak_memory_bandwidth_gbps (float | None)

• avg_quantize_latency_ms_per_token (float | None)

chip: str = 'unknown'

chip_generation: int = 0

total_memory_bytes: int | None = None

available_memory_bytes: int = 0

mlx_version: str | None = None

macos_version: str | None = None

metal_available: bool = True

peak_memory_bandwidth_gbps: float | None = None

avg_quantize_latency_ms_per_token: float | None = None

classmethod detect()

Build a HardwareProfile for the current machine (see detect_hardware_profile).

Return type:

HardwareProfile

property bandwidth_gbps: float | None

measured, else nominal, else None.

Type:

Likely peak bandwidth

to_dict()

Serialize all fields to a plain dict.

Return type:

dict

class veloxquant_mlx.ModelProfile(model_id='unknown', architecture='unknown', num_layers=0, num_query_heads=0, num_kv_heads=0, head_dim=0, attention_type='gqa', dtype='float16', parameter_count=None, hidden_size=None, _extra=<factory>)

Bases: object

Normalized model-architecture description used by the planner.

Variables:

• model_id – Model identifier (any string; the caller’s model_id).

• architecture – Short architecture slug ("llama", "qwen", …).

• num_layers – Number of attention-bearing layers.

• num_query_heads – Attention heads per layer.

• num_kv_heads – Key/value heads per layer (== num_query_heads for MHA).

• head_dim – Attention head dimension.

• attention_type – "mha", "gqa", or "mqa".

• dtype – Expected compute dtype ("float16", …).

• parameter_count – Total weight count, when the config reports one.

• hidden_size – Model hidden width, when known (informational).

Parameters:

• model_id (str)

• architecture (str)

• num_layers (int)

• num_query_heads (int)

• num_kv_heads (int)

• head_dim (int)

• attention_type (str)

• dtype (str)

• parameter_count (int | None)

• hidden_size (int | None)

• _extra (dict[str, Any])

model_id: str = 'unknown'

architecture: str = 'unknown'

num_layers: int = 0

num_query_heads: int = 0

num_kv_heads: int = 0

head_dim: int = 0

attention_type: str = 'gqa'

dtype: str = 'float16'

parameter_count: int | None = None

hidden_size: int | None = None

property baseline_kv_bytes_per_token: int

Full K+V footprint in bytes for one token, one layer (fp16).

baseline_kv_bytes(num_layers=None, batch=1)

Total fp16 K+V footprint across layers/batch for one token.

Return type:

int

Parameters:

• num_layers (int | None)

• batch (int)

to_dict()

Serialize all fields to a plain dict.

Return type:

dict[str, Any]

class veloxquant_mlx.WorkloadProfile(context_length=4096, generation_length=512, batch_size=1, num_concurrent_requests=1, objective='balanced', max_latency_ms=None)

Bases: object

Normalized description of a serving workload.

Variables:

• context_length – Maximum prompt context the served cache must hold.

• generation_length – Expected number of decode tokens per request.

• batch_size – Parallel requests sharing the KV cache.

• num_concurrent_requests – Requests served concurrently (drives how many caches the memory budget must cover).

• objective – One of the WorkloadObjective names; the planner uses this to weight the memory/latency/throughput/quality axes.

• max_latency_ms – Hard latency ceiling, when the caller knows one (used to exclude strategies whose analytical proxy exceeds it).

Parameters:

• context_length (int)

• generation_length (int)

• batch_size (int)

• num_concurrent_requests (int)

• objective (str)

• max_latency_ms (float | None)

context_length: int = 4096

generation_length: int = 512

batch_size: int = 1

num_concurrent_requests: int = 1

objective: str = 'balanced'

max_latency_ms: float | None = None

to_dict()

Serialize all fields to a plain dict.

Return type:

dict[str, Any]

property total_tokens_per_request: int

Context + generation, the cache lifetime of one request.

property effective_batch: int

Attention batch the memory estimator must budget for.

class veloxquant_mlx.WorkloadObjective

Bases: object

Identifier constants for the planner’s supported objectives.

Kept as a plain object (not an Enum) so downstream code can pass arbitrary objective names; the planner’s DEFAULT_OBJECTIVE_WEIGHTS only assigns weights to the canonical ones below and falls back to balanced for anything else.

MEMORY = 'memory'

LATENCY = 'latency'

THROUGHPUT = 'throughput'

QUALITY = 'quality'

BALANCED = 'balanced'

class veloxquant_mlx.RecommendationResult(objective, ranked, candidates, model, hardware, workload, fallback_used=False, fallback_reason=None)

Bases: object

Full outcome of a planning run, ready for the explainer / CLI.

Parameters:

• objective (str)

• ranked (list[ScoredMethod])

• candidates (CandidateFilterResult)

• model (ModelProfile)

• hardware (HardwareProfile)

• workload (WorkloadProfile)

• fallback_used (bool)

• fallback_reason (str | None)

objective: str

ranked: list[ScoredMethod]

candidates: CandidateFilterResult

model: ModelProfile

hardware: HardwareProfile

workload: WorkloadProfile

fallback_used: bool = False

fallback_reason: str | None = None

property best: ScoredMethod | None

Top-ranked method, or None if nothing survived filtering.

to_dict()

Serialize the full result (ranked list, candidates, inputs) to a plain dict.

Return type:

dict[str, Any]

class veloxquant_mlx.MemoryEstimate(method, baseline_bytes, compressed_bytes, workspace_bytes=0, peak_bytes=0, resident_bytes=0, confidence='high', assumptions=<factory>)

Bases: object

Breakdown of one method’s predicted KV-cache memory footprint.

Variables:

• method – Method name this estimate describes.

• baseline_bytes – fp16 K+V for the full context x batch (the no-compression starting point).

• compressed_bytes – Projected resident K+V after the method’s compression/eviction model.

• workspace_bytes – Predicted non-tensor overhead (codebooks, allocator).

• peak_bytes – Largest allocation during prefill (usually baseline).

• resident_bytes – Footprint that stays resident during decode.

• confidence – "high" for curated-model methods, "medium" for a generic compression default, "low" for unmodeled methods.

• assumptions – Human-readable list of what the model assumed.

Parameters:

• method (str)

• baseline_bytes (int)

• compressed_bytes (int)

• workspace_bytes (int)

• peak_bytes (int)

• resident_bytes (int)

• confidence (str)

• assumptions (list[str])

method: str

baseline_bytes: int

compressed_bytes: int

workspace_bytes: int = 0

peak_bytes: int = 0

resident_bytes: int = 0

confidence: str = 'high'

assumptions: list[str]

property reduction_ratio: float

Fraction of baseline the method is projected to keep (0..1).

property savings_percent: float

Percent of baseline bytes saved (0..100).

to_dict()

Serialize all fields (plus derived ratio/percent) to a plain dict.

Return type:

dict[str, Any]

class veloxquant_mlx.CacheRoutePlanner(n_shards, qcap, warm_slots_per_shard)

Bases: object

Builds a periodic RoutingTable from measured session rates.

Implements Algorithm 1 from the paper: size each admitted session’s shard count from its rate relative to qcap (Eq. 1), admit sessions in decreasing-rate order up to the warm-slot budget, then place admitted sessions with longest-processing-time-first (LPT) list scheduling so no shard’s expected load is far from the mean.

This planner is stateless between calls — call plan() once per control interval with fresh rates, and hold onto the returned RoutingTable for the whole interval rather than recomputing per request. Recomputing every interval, versus reusing a stale table, is a real tradeoff the paper measures directly (Appendix E): a fresh plan changes most owners’ shard assignments (high “cache churn”) even when only a few owners’ rates actually moved, which costs a rewarming dip; a stale plan drifts and loses some hit rate/latency instead. This class always recomputes fresh — callers wanting hysteresis should diff two RoutingTable instances themselves before switching.

Parameters:

• n_shards (int) – Number of pool shards to plan across (R in the paper). Use 1 if the pool isn’t partitioned — every admitted owner then simply gets warm-set priority with no placement decision to make.

• qcap (float) – Load cap per shard, in the same rate unit as SessionRate.rate (Section 3.1). Calibrate from a single shard’s measured latency/load knee, as the paper recommends — this module does not calibrate it for you.

• warm_slots_per_shard (float) – Warm-set capacity per shard, W in the paper. Total admission budget is n_shards * warm_slots_per_shard, counted in the same units as rate/qcap (equal-slot admission model; see the paper’s Section 3.1 and 6 for the heterogeneous-prefix-size limitation this inherits).

Example:

planner = CacheRoutePlanner(n_shards=4, qcap=50.0, warm_slots_per_shard=200.0)
table = planner.plan(estimator.rates())
shard = table.preferred_shard(owner, fallback=hash(owner) % planner.n_shards)

plan(rates)

Compute a fresh routing table from the given session rates.

Parameters:

rates (list[SessionRate]) – Estimated per-session rates for this control interval. Order does not matter; sessions are re-sorted internally by decreasing rate as the paper’s Algorithm 1 requires.

Return type:

RoutingTable

Returns:

A RoutingTable valid for the caller’s next control interval.

class veloxquant_mlx.RateEstimator(half_life=20.0, max_owners=None, _counts=<factory>, _last_seen=<factory>, _clock=0)

Bases: object

Tracks a per-owner exponentially-weighted request rate estimate.

A minimal helper for callers who don’t already track per-session rates: call record() on every incoming request, then rates() to get the SessionRate list a CacheRoutePlanner needs. Not required — any source of {owner: rate} works, including offline aggregate statistics as in the paper’s telemetry-derived workload.

Memory is bounded by the number of distinct owners tracked (one float in _counts and one int in _last_seen each), not by how many requests have been recorded — an owner’s whole contribution to memory is released by forget(). That bound is only enforced if the caller calls forget() on session close; nothing here does it automatically. A caller that can’t guarantee that — e.g. a long-running server seeing many short-lived session IDs — should set max_owners so _counts self-bounds instead of growing without limit.

Parameters:

• half_life (float) – Number of record() calls (from any owner) for an owner’s rate contribution to decay by half. Smaller values track bursts faster but are noisier; larger values are stable but slow to react to a session going cold or hot.

• max_owners (int | None) – Maximum distinct owners to track at once. None (the default) preserves the original unbounded behavior. When set, recording a request from a not-yet-tracked owner that would push the owner count past this cap first evicts the single currently-lowest-rate owner (an approximation of “the coldest session,” not a full LRU) to make room. Does not add a branch to the already-tracked-owner path, so the hot case (an owner already in _counts) is unaffected.

• _counts (dict[int, float])

• _last_seen (dict[int, int])

• _clock (int)

half_life: float = 20.0

max_owners: int | None = None

record(owner)

Record one request arrival for owner.

The estimate is an exponentially weighted count of the owner’s arrivals on a global clock that ticks once per recorded request (from any owner), so it is proportional to the owner’s arrival rate and decays while the owner is silent. Cost is one dict lookup, one pow and two dict writes for an already-tracked owner; the eviction scan only runs for a new owner when max_owners is set.

Return type:

None

Parameters:

owner (int)

rate(owner)

Current smoothed rate estimate for owner (0.0 if never seen).

Return type:

float

Parameters:

owner (int)

rates()

Snapshot every tracked owner’s current rate as a SessionRate list.

Return type:

list[SessionRate]

forget(owner)

Drop all tracked state for owner (e.g. on session close).

Return type:

None

Parameters:

owner (int)

class veloxquant_mlx.RoutingTable(shards, expected_load, n_shards, admitted_rate_total, unadmitted_rate_total)

Bases: object

A fixed admission-and-placement plan for one control interval.

Variables:

• shards – Maps each admitted owner to the tuple of shard ids it may use, ordered by preference (most-loaded-eligible-first is not guaranteed; callers needing a single preferred shard should use shards[owner][0]). An owner with more than one shard should split its load evenly across them, per the paper’s kb replication (Eq. 1) — an owner needs more than one shard only when its rate exceeds qcap on its own.

• expected_load – Per-shard expected load after this plan’s placement, indexed by shard id. Sums admitted owners’ rate / kb over every shard each owner was placed on, plus each shard’s cold-tail share of unadmitted traffic. For diagnostics and shadow-replay comparisons only — not enforced at dispatch time.

• n_shards – Total number of shards this plan was computed for.

• admitted_rate_total – Sum of rate over admitted owners, for computing what fraction of traffic the warm set actually covers.

• unadmitted_rate_total – Sum of rate over owners considered but not admitted (the plan’s cold-tail fallback traffic).

Parameters:

• shards (dict[int, tuple[int, ...]])

• expected_load (list[float])

• n_shards (int)

• admitted_rate_total (float)

• unadmitted_rate_total (float)

shards: dict[int, tuple[int, ...]]

expected_load: list[float]

n_shards: int

admitted_rate_total: float

unadmitted_rate_total: float

is_admitted(owner)

Whether owner has a warm-set placement in this plan.

Return type:

bool

Parameters:

owner (int)

preferred_shard(owner, fallback=0)

The single best shard for owner, or fallback if not admitted.

For an owner with kb > 1 shards, returns the currently least-loaded of its assigned shards (ties broken by lowest shard id), so repeated calls track load drift within the same interval even though the set of eligible shards stays fixed.

Return type:

int

Parameters:

• owner (int)

• fallback (int)

imbalance()

Ratio of the most-loaded shard’s expected load to the mean.

Matches the “imbalance” metric in the paper’s Table 4/5 (1.0x is perfectly balanced; higher is worse). Returns 1.0 for zero total load (nothing to balance).

Return type:

float

class veloxquant_mlx.SessionRate(owner, rate)

Bases: object

A session’s estimated request rate, in requests per unit time.

Variables:

• owner – Opaque session/owner id, matching BlockPoolAllocator owner ids.

• rate – Estimated request rate (lambda_b in the paper). Any consistent unit works as long as qcap uses the same unit.

Parameters:

• owner (int)

• rate (float)

owner: int

rate: float

Public API

The names below are re-exported at the top level for convenience; each is documented once, at its defining module (linked from the subpackage pages below), to avoid duplicate entries.

KVCacheBuilder()	Fluent builder for KVCache construction with validation.
KVCacheConfig([method, head_dim, ...])	Configuration for a KVCache instance.
KVCacheFactory()	Factory for creating KVCache instances from a KVCacheConfig.
ArtifactStore()	DAO interface for loading and saving precomputed quantization artifacts.
KVCache()	Abstract base class for KV cache implementations.
Quantizer()	Abstract base class for all vector quantizers.
QuantizationObserver()	Observer for pipeline events (timing, memory, distortion).
EncodedVector(quantizer_type, batch_size, dim)	Typed output of Quantizer.encode().
QuantizationContext(x_original, mode, x_current)	Mutable payload flowing through a QuantizationHandler chain.
TransformResult(angles, final_radius, n_levels)	Output of RecursivePolarTransform.forward().
ArtifactNotFoundError	Raised when a required precomputed artifact is missing from the store.
BlockPoolExhaustedError	Raised when a BlockPoolAllocator has no free blocks left to allocate.
CodebookDimensionMismatch	Raised when a codebook's shape does not match the expected dimension.
CyclicPipelineError	Raised when a QuantizationGraph contains a cycle.
QuantizerConfigError	Raised when a quantizer or KV cache is misconfigured.
BlockPoolAllocator([config])	Fixed-size block allocator for KV-cache storage.
PoolConfig([block_size, n_blocks, ...])	Configuration for a BlockPoolAllocator.
PooledKVCache(inner, pool[, owner, format])	Wraps any VeloxQuant KVCache with block-pool-backed memory accounting.
QuantizerFactory()	Factory for creating Quantizer instances via the QuantizerRegistry.
allocate_bits_ratequant(sensitivities, ...)	Allocate per-layer bit-widths via RateQuant Theorem 2.
calibrate_layer_sensitivities(model, tokenizer)	Run a calibration forward pass and return per-layer sensitivity.
fit_distortion_curve(head_dim[, ...])	Fit D(b) = α·β^(-b) on synthetic unit-norm Gaussian keys.
VecInferKVCache(config)	KV cache implementing VecInfer's dual-transform product VQ.
apply_dual_transform_keys(K, smooth, H)	Apply K_tilde = (K / lambda) @ H (smooth then Hadamard).
apply_dual_transform_queries(q, smooth, H)	Apply q_tilde = (q * lambda) @ H so q_tilde @ K_tilde.T == q @ K.T.
calibrate_smooth_factors(keys_calib[, eps])	Compute per-(head, channel) smooth scaling factor from a key sample.
train_codebook(x, n_centroids[, max_iter, seed])	Train a VQ codebook on flat sub-vector samples.
walsh_hadamard_matrix(d[, dtype])	Construct an orthonormal Walsh-Hadamard matrix.
KeyNormObserver()	Accumulates per-token key L2 norm² for later RateQuant allocation.
KeyNormReport(n_tokens, mean_norm_sq, ...)	Summary statistics produced by KeyNormObserver.
KVCacheProfiler(cache[, head_dim, layer_id])	Wraps a KVCache instance, recording per-call timing and memory.
MLXCacheProfiler(cache[, layer_id])	Wraps an mlx_lm.models.cache.KVCache-style cache's update_and_fetch.
LayerProfile(layer_id[, n_quantize_calls, ...])	Aggregated profiling stats for a single KV-cache layer.
ProfileReport([layers, elapsed_s])	Profiling results across one or more layers.
format_profile_table(report)	Render a ProfileReport as a human-readable fixed-width table.
profile_layers(profilers[, elapsed_s])	Aggregate a list of KVCacheProfiler instances into one ProfileReport.
AutoConfigResult(config, reason)	The selected configuration plus the reasoning behind it.
HardwareInfo([total_memory_bytes, ...])	Describes the machine the KV cache will run on.
WorkloadSpec([head_dim, seq_len, n_layers, ...])	Describes the job the KV cache needs to serve.
detect_hardware_info()	Auto-detect HardwareInfo from the running MLX device.
select_kv_cache_config(workload[, hardware])	Pick a KVCacheConfig for the given workload and hardware.
AutoOptimizer([options])	One-stop automatic KV-cache strategy recommender with caching and live validation.
AutoOptimizerOptions([benchmark_db_dir, ...])	Constructor options for AutoOptimizer.
HardwareProfile([chip, chip_generation, ...])	Comprehensive description of the Apple-Silicon environment.
ModelProfile([model_id, architecture, ...])	Normalized model-architecture description used by the planner.
WorkloadProfile([context_length, ...])	Normalized description of a serving workload.
WorkloadObjective()	Identifier constants for the planner's supported objectives.
RecommendationResult(objective, ranked, ...)	Full outcome of a planning run, ready for the explainer / CLI.
MemoryEstimate(method, baseline_bytes, ...)	Breakdown of one method's predicted KV-cache memory footprint.
CacheRoutePlanner(n_shards, qcap, ...)	Builds a periodic RoutingTable from measured session rates.
RateEstimator([half_life, max_owners, ...])	Tracks a per-owner exponentially-weighted request rate estimate.
RoutingTable(shards, expected_load, ...)	A fixed admission-and-placement plan for one control interval.
SessionRate(owner, rate)	A session's estimated request rate, in requests per unit time.

Subpackages

• veloxquant_mlx.allocators package
  • Submodules
    • veloxquant_mlx.allocators.kvtc_dp module
      • THE MECHANISM GAP THIS FILLS
      • DISTORTION MODEL — reused, not reinvented
      • dp_allocate_bits()
    • veloxquant_mlx.allocators.ratequant module
      • calibrate_layer_sensitivities()
      • fit_distortion_curve()
      • allocate_bits_ratequant()
    • veloxquant_mlx.allocators.vecinfer module
      • calibrate_smooth_factors()
      • walsh_hadamard_matrix()
      • apply_dual_transform_keys()
      • apply_dual_transform_queries()
      • train_codebook()
      • quantize_vq()
      • dequantize_vq()
      • compute_query_lut()
• veloxquant_mlx.artifacts package
  • Submodules
    • veloxquant_mlx.artifacts.base module
      • ArtifactStore
    • veloxquant_mlx.artifacts.memory_store module
      • InMemoryArtifactStore
    • veloxquant_mlx.artifacts.npy_store module
      • NpyArtifactStore
• veloxquant_mlx.benchmarks package
  • Submodules
    • veloxquant_mlx.benchmarks.attend_benchmark module
      • run()
      • main()
    • veloxquant_mlx.benchmarks.benchmark_db module
      • BenchmarkRecord
      • BenchmarkDatabase
      • benchmark_fingerprint()
      • default_benchmark_dir()
    • veloxquant_mlx.benchmarks.cacheroute_benchmark module
      • gini_coefficient()
      • SyntheticWorkload
      • generate_workload()
      • PolicyResult
      • run_benchmark()
      • format_summary_table()
      • results_to_json()
      • main()
    • veloxquant_mlx.benchmarks.comm_vq_benchmark module
    • veloxquant_mlx.benchmarks.falcon_rabitq_benchmark module
    • veloxquant_mlx.benchmarks.metal_kernel_benchmark module
    • veloxquant_mlx.benchmarks.model_kv_benchmark module
    • veloxquant_mlx.benchmarks.rabitq_benchmark module
    • veloxquant_mlx.benchmarks.workload_replay_benchmark module
      • WorkloadScenario
      • MemorySnapshot
      • WorkloadResult
      • run_workload()
      • run_suite()
      • format_summary_table()
      • results_to_json()
      • main()
• veloxquant_mlx.cache package
  • Submodules
    • veloxquant_mlx.cache.a2ats_cache module
      • A2ATSKVCache
    • veloxquant_mlx.cache.adakv_cache module
      • AdaKVCache
    • veloxquant_mlx.cache.age_tiered_cache module
      • AgeTieredKVCache
    • veloxquant_mlx.cache.amc_cache module
      • AMCKVCache
    • veloxquant_mlx.cache.anchorkv_cache module
      • AnchorKVKVCache
    • veloxquant_mlx.cache.base module
      • KVCacheConfig
      • KVCacheFactory
      • KVCacheBuilder
    • veloxquant_mlx.cache.cachegen_cache module
      • CacheGenKVCache
    • veloxquant_mlx.cache.cam_cache module
      • CaMKVCache
    • veloxquant_mlx.cache.chunkkv_cache module
      • ChunkKVCache
    • veloxquant_mlx.cache.chunkkv_coordinator module
      • ChunkKVIndexReuseCoordinator
    • veloxquant_mlx.cache.curdkv_cache module
      • CurDKVKVCache
    • veloxquant_mlx.cache.gear_cache module
      • GEARKVCache
    • veloxquant_mlx.cache.h2o_cache module
      • H2OKVCache
    • veloxquant_mlx.cache.keyformer_cache module
      • KeyformerKVCache
    • veloxquant_mlx.cache.kitty_cache module
      • KittyKVCache
    • veloxquant_mlx.cache.kivi_cache module
      • KIVIKVCache
    • veloxquant_mlx.cache.knorm_cache module
      • L2NormKVCache
    • veloxquant_mlx.cache.kvquant_cache module
      • KVQuantKVCache
    • veloxquant_mlx.cache.kvtc_cache module
      • NOT PATH-DEPENDENT (contrast with the eviction family)
      • Design
      • Byte accounting
      • KVTCKVCache
    • veloxquant_mlx.cache.kvzip_cache module
      • KVzipKVCache
    • veloxquant_mlx.cache.minicache_cache module
      • MiniCacheKVCache
    • veloxquant_mlx.cache.minicache_coordinator module
      • MiniCacheCoordinator
    • veloxquant_mlx.cache.morphkv_cache module
      • MorphKVKVCache
    • veloxquant_mlx.cache.nestedkv_cache module
      • NestedKVKVCache
    • veloxquant_mlx.cache.nsnquant_cache module
      • NSNQuantKVCache
    • veloxquant_mlx.cache.options module
      • VecInferOptions
      • KIVIOptions
      • SVDqOptions
      • KittyOptions
      • AdaKVOptions
      • XQuantOptions
      • KVQuantOptions
      • PALUOptions
      • CacheGenOptions
      • MiniCacheOptions
      • GEAROptions
      • ZipCacheOptions
      • SnapKVOptions
      • StreamingLLMOptions
      • H2OOptions
      • TOVAOptions
      • PyramidKVOptions
      • SqueezeAttentionOptions
      • ChunkKVOptions
      • CaMOptions
      • XKVOptions
      • NSNQuantOptions
      • L2NormOptions
      • SKVQOptions
      • QFiltersOptions
      • KeyformerOptions
      • MorphKVOptions
      • KVzipOptions
      • KVTCOptions
      • CurDKVOptions
      • NestedKVOptions
      • AMCOptions
      • A2ATSOptions
      • AnchorKVOptions
      • RocketKVOptions
      • AgeTieredKVOptions
      • SpectralQuantOptions
    • veloxquant_mlx.cache.palu_cache module
      • PALUKVCache
    • veloxquant_mlx.cache.polar_cache module
      • PolarQuantKVCache
    • veloxquant_mlx.cache.pyramidkv_cache module
      • PyramidKVCache
    • veloxquant_mlx.cache.qfilters_cache module
      • TWO FILTER SOURCES
      • QFiltersKVCache
    • veloxquant_mlx.cache.qjl_cache module
      • QJLKVCache
    • veloxquant_mlx.cache.registry module
      • ServeTier
      • Maturity
      • MethodFamily
      • MethodInfo
      • StrategyCapabilities
      • all_method_names()
      • list_methods()
      • get_method()
      • static_method_info()
      • probe_serve_tier()
      • describe_field()
      • field_is_relevant()
      • telemetry_coverage()
      • TelemetryCoverage
      • DEFAULT_SERVE_METHOD
    • veloxquant_mlx.cache.rocketkv_cache module
      • RocketKVKVCache
    • veloxquant_mlx.cache.sink_cache module
      • SinkProtectedKVCache
    • veloxquant_mlx.cache.skvq_cache module
      • SKVQKVCache
    • veloxquant_mlx.cache.sliding_window_cache module
      • SlidingWindowKVCache
    • veloxquant_mlx.cache.snapkv_cache module
      • SnapKVKVCache
    • veloxquant_mlx.cache.spectral_cache module
      • SpectralQuantKVCache
    • veloxquant_mlx.cache.squeeze_cache module
      • SqueezeAttentionCache
    • veloxquant_mlx.cache.squeeze_coordinator module
      • SqueezeCoordinator
    • veloxquant_mlx.cache.streaming_llm_cache module
      • StreamingLLMKVCache
    • veloxquant_mlx.cache.svdq_cache module
      • SVDqKVCache
    • veloxquant_mlx.cache.tova_cache module
      • TOVAKVCache
    • veloxquant_mlx.cache.turboquant_cache module
      • TurboQuantKVCache
    • veloxquant_mlx.cache.turboquant_rvq_cache module
      • TurboQuantRVQKVCache
    • veloxquant_mlx.cache.vecinfer_cache module
      • VecInferKVCache
    • veloxquant_mlx.cache.xkv_cache module
      • XKVCache
    • veloxquant_mlx.cache.xkv_coordinator module
      • XKVCoordinator
    • veloxquant_mlx.cache.xquant_cache module
      • XQuantKVCache
    • veloxquant_mlx.cache.xquant_coordinator module
      • XQuantCoordinator
    • veloxquant_mlx.cache.zipcache_cache module
      • ZipCacheKVCache
• veloxquant_mlx.cli package
  • Submodules
    • veloxquant_mlx.cli.auto_config module
      • main()
    • veloxquant_mlx.cli.benchmark module
      • main()
    • veloxquant_mlx.cli.estimate_memory module
      • main()
    • veloxquant_mlx.cli.methods module
      • build_parser()
      • main()
    • veloxquant_mlx.cli.panel module
      • build_parser()
      • main()
    • veloxquant_mlx.cli.precompute module
      • main()
    • veloxquant_mlx.cli.profile module
      • ACCOUNTING_NOTE
      • build_parser()
      • validate_method()
      • parse_overrides()
      • build_config()
      • run_profile()
      • main()
    • veloxquant_mlx.cli.profile_hardware module
      • main()
    • veloxquant_mlx.cli.recommend module
      • main()
    • veloxquant_mlx.cli.serve module
      • READY_PREFIX
      • build_parser()
      • parse_overrides()
      • validate_method()
      • build_config()
      • attach_cache()
      • emit_ready()
      • run_server()
      • main()
    • veloxquant_mlx.cli.worker module
      • main()
• veloxquant_mlx.codebooks package
  • Submodules
    • veloxquant_mlx.codebooks.adaptive_codebook module
      • AdaptiveScalarCodebook
    • veloxquant_mlx.codebooks.base module
      • CodebookFactory
    • veloxquant_mlx.codebooks.precompute module
      • precompute()
      • main()
    • veloxquant_mlx.codebooks.scalar_codebook module
      • ScalarCodebook
    • veloxquant_mlx.codebooks.strategies module
      • LloydMaxGaussianStrategy
      • LloydMaxBetaStrategy
      • PolarAngleSamplingStrategy
      • UniformStrategy
• veloxquant_mlx.config package
  • Submodules
    • veloxquant_mlx.config.auto_config module
      • WorkloadSpec
      • HardwareInfo
      • AutoConfigResult
      • detect_hardware_info()
      • select_kv_cache_config()
• veloxquant_mlx.core package
  • Submodules
    • veloxquant_mlx.core.abstractions module
      • Quantizer
      • Preconditioner
      • Codebook
      • KVCache
      • Transform
      • QuantizationHandler
      • InnerProductStrategy
      • CodebookStrategy
      • ArtifactStore
      • QuantizationObserver
    • veloxquant_mlx.core.constants module
    • veloxquant_mlx.core.context module
      • QuantizationContext
      • EncodedVector
      • TransformResult
    • veloxquant_mlx.core.exceptions module
      • QuantizerConfigError
      • ArtifactNotFoundError
      • CyclicPipelineError
      • CodebookDimensionMismatch
      • BlockPoolExhaustedError
      • OwnerAlreadyActiveError
    • veloxquant_mlx.core.registry module
      • QuantizerRegistry
      • CodebookRegistry
      • PreconditionerRegistry
• veloxquant_mlx.dsa package
  • Submodules
    • veloxquant_mlx.dsa.avl_tree module
      • AVLNode
      • AVLTree
      • VoronoiTree
    • veloxquant_mlx.dsa.bit_pack module
      • BitPackBuffer
    • veloxquant_mlx.dsa.dag module
      • QuantizationGraph
    • veloxquant_mlx.dsa.heap module
      • MaxHeap
      • SortedChannelIndex
    • veloxquant_mlx.dsa.ring_buffer module
      • RingBuffer
• veloxquant_mlx.handlers package
  • Submodules
    • veloxquant_mlx.handlers.base module
      • QuantizationHandler
      • QuantizationContext
    • veloxquant_mlx.handlers.bit_pack_handler module
      • BitPackingHandler
    • veloxquant_mlx.handlers.normalization module
      • NormalizationHandler
    • veloxquant_mlx.handlers.outlier_split module
      • OutlierSplitHandler
    • veloxquant_mlx.handlers.polar_handler module
      • PolarTransformHandler
    • veloxquant_mlx.handlers.qjl_residual_handler module
      • QJLResidualHandler
    • veloxquant_mlx.handlers.rotation_handler module
      • RotationHandler
    • veloxquant_mlx.handlers.scalar_quant_handler module
      • ScalarQuantizerHandler
    • veloxquant_mlx.handlers.value_quant_handler module
      • ValueQuantizerHandler
• veloxquant_mlx.integration package
  • Submodules
    • veloxquant_mlx.integration.mlx_lm_patch module
      • patch_model_kv_cache()
    • veloxquant_mlx.integration.mlx_vlm_patch module
      • patch_vlm_kv_cache()
    • veloxquant_mlx.integration.prefix_cache module
      • PrefixCache
• veloxquant_mlx.math package
  • Submodules
    • veloxquant_mlx.math.distributions module
      • beta_pdf()
      • polar_angle_pdf()
      • gaussian_pdf()
    • veloxquant_mlx.math.lloyd_max module
      • lloyd_max()
    • veloxquant_mlx.math.rotation module
      • is_hadamard_compatible()
      • make_hadamard_diagonal()
      • make_rotation_matrix()
      • make_jl_matrix()
• veloxquant_mlx.memory package
  • Submodules
    • veloxquant_mlx.memory.block_pool module
      • PoolConfig
      • Block
      • AllocationStats
      • BlockPoolAllocator
    • veloxquant_mlx.memory.mlx_storage module
      • MLXBlockStorage
    • veloxquant_mlx.memory.pool_backed_cache module
      • PoolBackedKVCache
      • build_pooled_caches()
    • veloxquant_mlx.memory.pooled_cache module
      • PooledKVCache
• veloxquant_mlx.metal package
  • Submodules
    • veloxquant_mlx.metal.fused_sdpa module
      • Design
      • Memory model
      • Throughput
      • metal_fused_sdpa()
      • supports_shape()
      • patch_mlx_lm_for_fused_sdpa()
      • unpatch_mlx_lm()
      • is_patched()
    • veloxquant_mlx.metal.kernels module
      • tova_fused_evict()
      • tova_fused_evict_virtual_values()
      • tova_fused_evict_indices()
      • crosskv_rope_recode()
      • vecinfer_dequant_metal()
      • vecinfer_quantize_metal()
      • vecinfer_encode_decode_metal()
      • vecinfer_encode_decode_simple_metal()
      • turboquant_bit_pack()
      • turboquant_bit_unpack()
      • turboquant_scalar_quantize()
      • turboquant_scalar_dequantize()
      • turboquant_hadamard_quantize()
      • scalar_fused_decode_attend()
      • scalar_fused_decode_attend_batched()
      • kivi_group_quant_dequant()
      • qjl_encode()
      • qjl_inner_product()
      • turboquant_fused_rvq_decode_attend()
      • rvq_quant_pack()
      • rvq_unpack_decode()
      • comm_vq_decode_metal()
      • rabitq_hamming_score()
      • rabitq_fused_attend()
      • rabitq_encode()
      • rabitq_pack_values()
      • rabitq_prefill_attend()
      • flash_prefill_attend()
      • streaming_prefill_attend()
      • h2o_fused_evict()
      • pyramidkv_fused_evict()
      • keyformer_fused_evict()
      • qfilters_fused_evict()
      • qfilters_score()
• veloxquant_mlx.observers package
  • Submodules
    • veloxquant_mlx.observers.base module
      • QuantizationEvent
    • veloxquant_mlx.observers.distortion module
      • DistortionReport
      • DistortionObserver
    • veloxquant_mlx.observers.key_norm module
      • KeyNormReport
      • KeyNormObserver
    • veloxquant_mlx.observers.latency module
      • LatencyObserver
    • veloxquant_mlx.observers.memory module
      • MemoryObserver
• veloxquant_mlx.outlier package
  • Submodules
    • veloxquant_mlx.outlier.detector module
      • OutlierDetector
• veloxquant_mlx.planning package
  • Submodules
    • veloxquant_mlx.planning.candidate_filter module
      • CandidateFilterResult
      • CandidateFilterOptions
      • filter_candidates()
    • veloxquant_mlx.planning.explainer module
      • explain()
      • explain_filtering()
      • explain_hardware_model()
    • veloxquant_mlx.planning.memory_estimator module
      • MemoryEstimate
      • MemoryEstimateError
      • estimate_memory()
      • estimate_candidate_memory()
      • method_quant_bits()
    • veloxquant_mlx.planning.strategy_planner module
      • ScoredMethod
      • RecommendationResult
      • PlanningOptions
      • DEFAULT_OBJECTIVE_WEIGHTS
      • plan_strategy()
      • recommend_strategy()
    • veloxquant_mlx.planning.workload module
      • WorkloadProfile
      • WorkloadObjective
      • workload_from_dict()
• veloxquant_mlx.preconditioners package
  • Submodules
    • veloxquant_mlx.preconditioners.base module
      • PreconditionerFactory
    • veloxquant_mlx.preconditioners.jl_sketch module
      • JLSketchPreconditioner
      • QJLEncoder
    • veloxquant_mlx.preconditioners.rotation module
      • HadamardPreconditioner
      • RotationPreconditioner
• veloxquant_mlx.profiling package
  • Submodules
    • veloxquant_mlx.profiling.hardware_profiler module
      • chip_generation()
      • HardwareProfile
      • detect_hardware_profile()
      • measure_bandwidth_gbps()
    • veloxquant_mlx.profiling.kv_profiler module
      • LayerProfile
      • ProfileReport
      • KVCacheProfiler
      • MLXCacheProfiler
      • profile_layers()
      • format_profile_table()
    • veloxquant_mlx.profiling.model_profiler module
      • ModelProfile
      • profile_model_from_config()
      • attention_type_from_heads()
      • architecture_alias()
• veloxquant_mlx.quantizers package
  • Submodules
    • veloxquant_mlx.quantizers.a2ats module
      • a2ats_query_second_moment()
      • a2ats_cholesky_factor()
      • a2ats_h_weighted_assignment()
      • a2ats_query_aware_assignment()
      • a2ats_select_retrieval_set()
    • veloxquant_mlx.quantizers.a2ats_rope module
      • rope_freqs_from_scaling()
      • a2ats_apply_exact_rope()
      • a2ats_apply_windowed_rope()
      • a2ats_apply_far_query_rope()
      • rope_remap_positions()
    • veloxquant_mlx.quantizers.adakv module
      • compute_head_norm_variance()
      • compute_head_attention_entropy()
      • allocate_head_bits()
      • quantize_head()
      • quantize_heads_batched()
    • veloxquant_mlx.quantizers.age_tiered module
      • AgeTierConfig
      • default_age_tiers()
      • assign_age_tiers()
      • age_tier_quantize()
      • age_tier_quantize_batched()
      • age_tiered_bytes()
      • full_fp16_bytes()
    • veloxquant_mlx.quantizers.amc module
      • AMCTierConfig
      • AMCThresholdState
      • init_amc_threshold_state()
      • amc_saliency()
      • amc_query_aware_saliency()
      • amc_query_aware_saliency_batched()
      • amc_assign_tiers()
      • amc_assign_tiers_batched()
      • amc_adaptive_thresholds()
      • amc_apply_rank_mask()
      • amc_quantize_tier()
      • amc_compress_tokens_batched()
      • amc_pack_low_tier()
      • amc_fp16_bytes()
      • full_amc_fp16_bytes()
    • veloxquant_mlx.quantizers.amc_calibration module
      • amc_calibrate_channel_order()
      • amc_permute_weights()
    • veloxquant_mlx.quantizers.anchorkv module
      • AnchorAssignment
      • ResidualCodec
      • select_anchors()
      • select_anchors_batched()
      • assign_and_project()
      • assign_and_project_batched()
      • pack_codes()
      • unpack_codes()
      • key_value_utility()
      • key_value_utility_batched()
      • allocate_residual_budget()
      • anchorkv_budget_slots()
    • veloxquant_mlx.quantizers.base module
      • QuantizerFactory
    • veloxquant_mlx.quantizers.cachegen module
      • CodeStream
      • quantize_to_codes()
      • dequant_codes()
      • token_delta()
      • symbol_entropy_bits()
      • entropy_coded_bytes()
      • fixed_width_bytes()
      • layer_group_bits()
      • cachegen_quant_dequant()
      • quantize_to_codes_batched()
      • dequant_codes_batched()
      • token_delta_batched()
      • entropy_coded_bytes_batched()
      • fixed_width_bytes_batched()
    • veloxquant_mlx.quantizers.cam module
      • Public API
      • most_similar_survivor()
      • merge_pair()
      • merge_gate_probability()
      • sample_merge_gate()
      • CaMState
      • init_cam_state()
      • cam_update()
      • cam_update_batched()
      • cam_get_kv()
      • cam_fp16_bytes()
      • full_cam_fp16_bytes()
    • veloxquant_mlx.quantizers.chunkkv module
      • Public API
      • chunk_partition()
      • chunk_scores()
      • chunkkv_keep_mask()
      • ChunkKVState
      • init_chunkkv_state()
      • chunkkv_update()
      • chunkkv_update_batched()
      • chunkkv_apply_reuse_indices()
      • chunkkv_apply_reuse_indices_batched()
      • chunkkv_trim_to()
      • chunkkv_trim_batched()
      • chunkkv_get_kv()
      • chunkkv_fp16_bytes()
      • full_chunkkv_fp16_bytes()
    • veloxquant_mlx.quantizers.comm_vq module
      • CommVQQuantizer
    • veloxquant_mlx.quantizers.composite module
      • CompositeQuantizer
    • veloxquant_mlx.quantizers.curdkv module
      • Public API
      • CurDKVState
      • init_curdkv_state()
      • curdkv_update()
      • curdkv_update_batched()
      • curdkv_get_kv()
      • curdkv_fp16_bytes()
      • full_curdkv_fp16_bytes()
    • veloxquant_mlx.quantizers.gear module
      • GEARState
      • quantize_base()
      • residual()
      • lowrank_error()
      • sparse_outliers()
      • sparse_outliers_batched()
      • gear_compress()
      • gear_reconstruct()
      • gear_reconstruct_batched()
      • gear_bytes()
      • base_only_bytes()
      • gear_quant_dequant()
    • veloxquant_mlx.quantizers.h2o module
      • Public API
      • H2OState
      • init_h2o_state()
      • h2o_update()
      • h2o_update_batched()
      • h2o_get_kv()
      • h2o_fp16_bytes()
      • full_h2o_fp16_bytes()
    • veloxquant_mlx.quantizers.keyformer module
      • WHERE THIS SITS IN THE REPO
      • THE HONESTY CRUX (read before trusting any number)
      • Public API (mirrors quantizers/h2o.py)
      • KeyformerState
      • init_keyformer_state()
      • keyformer_update()
      • keyformer_update_batched()
      • keyformer_get_kv()
      • keyformer_fp16_bytes()
      • full_keyformer_fp16_bytes()
    • veloxquant_mlx.quantizers.kitty module
      • rank_channels_by_sensitivity()
      • quantize_mixed_channels()
      • compute_running_variance()
      • hi_mask_from_variance_batched()
      • quantize_mixed_channels_batched()
    • veloxquant_mlx.quantizers.kivi module
      • KIVIQuantizer
    • veloxquant_mlx.quantizers.knorm module
      • Public API (mirrors quantizers/h2o.py)
      • KnormState
      • init_knorm_state()
      • knorm_update()
      • knorm_update_batched()
      • knorm_get_kv()
      • knorm_fp16_bytes()
      • full_knorm_fp16_bytes()
    • veloxquant_mlx.quantizers.kvquant module
      • DenseSparse
      • split_dense_sparse()
      • split_dense_sparse_batched()
      • fit_nuq_levels()
      • fit_nuq_levels_batched()
      • quantize_nuq()
      • quantize_nuq_batched()
      • dequant_nuq()
      • dequant_nuq_batched()
      • nuq_quant_dequant()
      • nuq_distortion()
    • veloxquant_mlx.quantizers.kvtc module
      • THE HONESTY CRUX (read before trusting any number)
      • Algorithm
      • Entropy-coding fallback (#77)
      • KVTCArtifact
      • kvtc_compress()
      • kvtc_compress_from_pca_fit()
      • kvtc_local_pca_batched()
      • kvtc_decompress()
      • kvtc_fp16_bytes()
      • kvtc_pre_entropy_bytes()
      • quantize_component()
    • veloxquant_mlx.quantizers.kvzip module
      • WHERE THIS SITS IN THE REPO
      • THE HONESTY CRUX (read before trusting any number)
      • Public API (mirrors quantizers/morphkv.py)
      • KVzipState
      • init_kvzip_state()
      • kvzip_update()
      • kvzip_update_batched()
      • kvzip_get_kv()
      • kvzip_fp16_bytes()
      • full_kvzip_fp16_bytes()
    • veloxquant_mlx.quantizers.minicache module
      • pair_layers_depth()
      • to_mag_dir()
      • slerp()
      • MergeResult
      • merge_pair()
      • reconstruct_layer()
      • merge_similarity()
    • veloxquant_mlx.quantizers.morphkv module
      • WHERE THIS SITS IN THE REPO
      • THE HONESTY CRUX (read before trusting any number)
      • Public API (mirrors quantizers/keyformer.py)
      • MorphKVState
      • init_morphkv_state()
      • morphkv_update()
      • morphkv_update_batched()
      • morphkv_get_kv()
      • morphkv_fp16_bytes()
      • full_morphkv_fp16_bytes()
    • veloxquant_mlx.quantizers.nestedkv module
      • Public API
      • NestedKVState
      • init_nestedkv_state()
      • block_size_for()
      • per_scale_anomaly_scores()
      • head_adaptive_blend()
      • surprise_gated_score()
      • nestedkv_score()
      • nestedkv_allocate_head_budgets()
      • nestedkv_compress_prefill()
      • nestedkv_append_decode()
      • nestedkv_get_kv()
      • nestedkv_fp16_bytes()
      • full_nestedkv_fp16_bytes()
      • per_scale_anomaly_scores_batched()
      • head_adaptive_blend_batched()
      • surprise_gated_score_batched()
      • nestedkv_score_batched()
      • nestedkv_compress_prefill_batched()
      • nestedkv_append_decode_batched()
    • veloxquant_mlx.quantizers.nsnquant module
      • nsn_transform()
      • nsn_inverse()
      • build_universal_codebook()
      • vq_encode()
      • vq_decode()
      • hadamard_forward()
      • hadamard_inverse()
    • veloxquant_mlx.quantizers.palu module
      • head_group_bounds()
      • group_head_svd()
      • project_to_latent()
      • reconstruct_from_latent()
      • quantize_latent()
      • project_to_latent_batched()
      • reconstruct_from_latent_batched()
      • quantize_latent_batched()
    • veloxquant_mlx.quantizers.polarquant module
      • PolarQuantizer
    • veloxquant_mlx.quantizers.pyramidkv module
      • Public API
      • pyramid_budgets()
      • PyramidState
      • init_pyramid_state()
      • pyramid_update()
      • pyramid_get_kv()
      • pyramid_fp16_bytes()
      • full_pyramid_fp16_bytes()
    • veloxquant_mlx.quantizers.qfilters module
      • TWO FILTER SOURCES (which one you get changes the guarantees)
      • Path-dependence applies to the FALLBACK ONLY
      • Public API (mirrors quantizers/knorm.py)
      • QFiltersState
      • init_qfilters_state()
      • estimate_filter_dir()
      • qfilters_update()
      • qfilters_update_batched()
      • qfilters_get_kv()
      • qfilters_fp16_bytes()
      • full_qfilters_fp16_bytes()
    • veloxquant_mlx.quantizers.qfilters_calibration module
      • Why this module has to exist
      • What this module computes (paper §3.2, step 1)
      • Public API
      • QFiltersCalibration
      • average_gqa_filters()
      • collect_query_activations()
      • compute_qfilters()
      • load_qfilters()
      • save_qfilters()
    • veloxquant_mlx.quantizers.qjl module
      • QJLQuantizer
    • veloxquant_mlx.quantizers.rabitq module
      • RaBitQQuantizer
    • veloxquant_mlx.quantizers.rocketkv module
      • PagedKeySummary
      • build_paged_summary()
      • append_paged_summary()
      • head_dim_topk_mask()
      • hsa_approx_scores()
      • select_topk_pages()
      • gather_page_tokens()
      • split_compression_ratio()
      • split_hsa_dims()
      • hsa_fp32_bytes()
      • build_paged_summary_batched()
      • append_paged_summary_batched()
    • veloxquant_mlx.quantizers.skvq module
      • channel_permutation()
      • invert_permutation()
      • apply_permutation()
      • clipped_group_quant()
      • clipped_group_dequant()
      • skvq_round_trip()
      • skvq_compressed_bytes()
      • skvq_fp16_bytes()
    • veloxquant_mlx.quantizers.snapkv module
      • SnapKVState
      • obs_window_attention_scores()
      • snap_select_indices()
      • snapkv_compress()
      • snapkv_fp16_bytes()
      • full_fp16_bytes()
    • veloxquant_mlx.quantizers.squeeze module
      • Public API
      • concentration_score()
      • squeeze_budgets()
      • SqueezeState
      • init_squeeze_state()
      • squeeze_update()
      • squeeze_update_batched()
      • squeeze_trim_batched()
      • squeeze_get_kv()
      • squeeze_fp16_bytes()
      • full_squeeze_fp16_bytes()
    • veloxquant_mlx.quantizers.streaming_llm module
      • StreamingWindow
      • init_streaming_window()
      • stream_update()
      • stream_get_kv()
      • stream_fp16_bytes()
      • full_stream_fp16_bytes()
    • veloxquant_mlx.quantizers.svdq module
      • svd_compress_keys()
      • svd_compress_keys_batched()
      • quantize_latents_mixed()
      • reconstruct_keys()
      • project_quantize_reconstruct_batched()
      • latent_group_slices()
      • equivalent_bit_width()
      • min_safe_rank()
      • MIN_SAFE_CHANNELS_PER_GROUP
      • DEFAULT_BIT_SCHEDULE
    • veloxquant_mlx.quantizers.tova module
      • Public API
      • TovaState
      • init_tova_state()
      • tova_update()
      • tova_get_kv()
      • tova_fp16_bytes()
      • full_tova_fp16_bytes()
    • veloxquant_mlx.quantizers.turboquant_mse module
      • TurboQuantMSE
    • veloxquant_mlx.quantizers.turboquant_prod module
      • TurboQuantProd
      • TurboQuantProdAdaptive
    • veloxquant_mlx.quantizers.turboquant_rvq module
      • TurboQuantRVQ
    • veloxquant_mlx.quantizers.xkv module
      • pair_layers_grouped()
      • joint_svd_compress()
      • project_into_shared_basis()
      • reconstruct_from_shared_basis()
      • quantize_latents_uniform()
      • project_quantize_reconstruct_batched()
    • veloxquant_mlx.quantizers.xquant module
      • GroupParams
      • pair_layers()
      • quantize_codes()
      • compute_reuse_params()
      • dequant_with_params()
      • quantize_residual()
      • quantize_codes_batched()
      • compute_reuse_params_batched()
      • dequant_with_params_batched()
      • quantize_residual_batched()
      • cross_layer_similarity()
    • veloxquant_mlx.quantizers.zipcache module
      • ZipCacheState
      • token_key_norms()
      • saliency_mask()
      • channel_quant()
      • channel_dequant()
      • zipcache_compress()
      • zipcache_reconstruct()
      • zipcache_bytes()
      • base_only_bytes()
      • zipcache_quant_dequant()
• veloxquant_mlx.routing package
  • Submodules
    • veloxquant_mlx.routing.cacheroute module
      • SessionRate
      • RoutingTable
      • CacheRoutePlanner
      • RateEstimator
• veloxquant_mlx.spectral package
  • Submodules
    • veloxquant_mlx.spectral.bit_allocator module
      • water_fill_bits()
    • veloxquant_mlx.spectral.calibrate module
      • load_cached_rotations()
      • save_rotations()
      • collect_kv_vectors_mlx()
      • calibrate_spectral_rotation()
      • calibrate_from_vectors()
    • veloxquant_mlx.spectral.participation_ratio module
      • compute_participation_ratio()
      • compute_spectral_gap()
    • veloxquant_mlx.spectral.spectral_quant module
      • SpectralQuantizer
• veloxquant_mlx.tools package
  • Submodules
    • veloxquant_mlx.tools.mac_recommender module
      • RecommendRequest
      • RecommendResult
      • estimate_kv_fp16_mb()
      • recommend()
      • ruleset_dict()
• veloxquant_mlx.transfer package
  • Submodules
    • veloxquant_mlx.transfer.apply module
      • load_mapper()
      • transfer_layer()
      • transfer_cache()
    • veloxquant_mlx.transfer.fit module
      • fit_ridge()
      • layer_r2()
      • select_source_layers()
      • fit_mapper()
    • veloxquant_mlx.transfer.mapper module
      • MatchedKVError
      • ModelKVSpec
      • MapperConfig
      • LayerMap
      • CrossModelMapper
    • veloxquant_mlx.transfer.rope module
      • rope_angles()
      • strip_rope()
      • apply_rope()
      • recode_rope()
• veloxquant_mlx.transforms package
  • Submodules
    • veloxquant_mlx.transforms.base module
      • Transform
    • veloxquant_mlx.transforms.polar module
      • RecursivePolarTransform
• veloxquant_mlx.ui package
  • Submodules
    • veloxquant_mlx.ui.config module
      • load_config()
      • save_config()
    • veloxquant_mlx.ui.memory module
      • memory_report()
    • veloxquant_mlx.ui.models module
      • local_models()
      • search_hub_models()
    • veloxquant_mlx.ui.server module
      • CONTROL_HOST
      • MAX_BODY_BYTES
      • PanelHandler
      • serve_panel()
    • veloxquant_mlx.ui.supervisor module
      • LOG_CAPACITY
      • LogLine
      • ServerSupervisor
• veloxquant_mlx.weight package
  • Submodules
    • veloxquant_mlx.weight.model_quantizer module
      • quantize_model()
      • compression_report()
    • veloxquant_mlx.weight.quantized_linear module
      • QuantizedLinear
    • veloxquant_mlx.weight.reservoir module
      • save_reservoir()
      • load_reservoir()
      • graft_reservoir()