veloxquant_mlx.tools package

Utility helpers that do not require MLX at import time.

class veloxquant_mlx.tools.RecommendRequest(chip, ram_gb, model_class, goal, seq_len=4096, n_layers=32, n_kv_heads=8, head_dim=128)[source]

Bases: object

Inputs describing the Mac and the model the user wants to run.

Variables:

  • chip – Apple Silicon chip family.

  • ram_gb – Total unified memory in GB (must be one of ALLOWED_RAM_GB).

  • model_class – Approximate parameter-count bucket of the model.

  • goal – Which recommendation heuristic to apply.

  • seq_len – Target sequence length used for the KV-cache size estimate.

  • n_layers – Number of attention layers.

  • n_kv_heads – Key/value head count.

  • head_dim – Attention head dimension.

Parameters:

  • chip (Literal['M1', 'M2', 'M3', 'M4'])

  • ram_gb (int)

  • model_class (Literal['1B', '3B', '7B', '14B', '32B', '70B', '120B', '235B', '671B'])

  • goal (Literal['everyday', 'max_key_accounting', 'max_context', 'best_quality', 'constant_memory'])

  • seq_len (int)

  • n_layers (int)

  • n_kv_heads (int)

  • head_dim (int)

chip: Literal['M1', 'M2', 'M3', 'M4']
ram_gb: int
model_class: Literal['1B', '3B', '7B', '14B', '32B', '70B', '120B', '235B', '671B']
goal: Literal['everyday', 'max_key_accounting', 'max_context', 'best_quality', 'constant_memory']
seq_len: int = 4096
n_layers: int = 32
n_kv_heads: int = 8
head_dim: int = 128
class veloxquant_mlx.tools.RecommendResult(method, knobs, key_accounting_ratio, resident_savings_likely, kv_fp16_mb, kv_compressed_mb_estimate, warnings, rationale)[source]

Bases: object

A transparent method recommendation with its rationale and warnings.

Variables:

  • method – Recommended KV-cache method name.

  • knobs – Suggested configuration for that method.

  • key_accounting_ratio – Estimated key-cache compression ratio.

  • resident_savings_likely – Whether the method is expected to actually free RAM (vs. only reporting a smaller accounted size).

  • kv_fp16_mb – Uncompressed fp16 KV-cache size, in MB.

  • kv_compressed_mb_estimate – Estimated compressed KV-cache size, in MB.

  • warnings – Human-readable caveats about fit or behavior.

  • rationale – Human-readable explanation for the pick.

Parameters:
method: str
knobs: dict[str, Any]
key_accounting_ratio: float
resident_savings_likely: bool
kv_fp16_mb: float
kv_compressed_mb_estimate: float
warnings: list[str]
rationale: str
to_dict()[source]

Serialize all fields to a plain dict.

Return type:

dict[str, Any]

veloxquant_mlx.tools.estimate_kv_fp16_mb(n_layers, n_kv_heads, head_dim, seq_len)[source]

Full K+V fp16 cache size in megabytes.

Return type:

float

Parameters:

  • n_layers (int)

  • n_kv_heads (int)

  • head_dim (int)

  • seq_len (int)

veloxquant_mlx.tools.recommend(req)[source]

Return a transparent method recommendation for Apple Silicon.

Return type:

RecommendResult

Parameters:

req (RecommendRequest)

veloxquant_mlx.tools.ruleset_dict()[source]

Export static metadata for docs / JS widgets.

Return type:

dict[str, Any]

Submodules