veloxquant_mlx.benchmarks.benchmark_db module

Persistent benchmark result store for the auto-selection pipeline.

Records measured KV-cache outcomes (memory, throughput, latency, quality delta) per (method, model, workload, environment) and lets the planner override its analytical estimates with real numbers when a close match exists. Storage is a plain JSON directory (an index.json plus one file per record) under ~/.cache/veloxquant/benchmarks/ so it is diff-able, debuggable, and portable. No record is ever partially written: files are atomically renamed.

class veloxquant_mlx.benchmarks.benchmark_db.BenchmarkRecord(id, method, model_id, architecture='unknown', context_length=4096, batch_size=1, memory_bytes=0, memory_reduction=None, throughput_tok_s=0.0, latency_ms_per_token=0.0, perplexity_delta=None, chip='unknown', mlx_version='unknown', macos_version='unknown', workload=<factory>, created_at=<factory>, source='manual', tags=<factory>)

Bases: object

One measured KV-cache outcome, keyed by benchmark_fingerprint().

Variables:

• method – Name of the KV-cache strategy measured.

• model_id – Model identifier the measurement was taken on.

• architecture – Architecture slug ("llama", …) for fuzzy matching.

• context_length – Context length the run used.

• batch_size – Attention batch the run used.

• memory_bytes – Measured resident KV-cache footprint in bytes.

• memory_reduction – Measured ratio of memory_bytes to the fp16 baseline at the same context/batch (0..1; 0.19 = 81% savings). None means not measured (distinct from 1.0, which is a measured “no savings”), so the planner keeps its analytic estimate.

• throughput_tok_s – Decode throughput in tokens/second.

• latency_ms_per_token – Mean per-token latency in milliseconds.

• perplexity_delta – PPL change vs fp16 baseline, when measured (None = n/a).

• chip – Apple-Silicon chip measured on ("M4").

• mlx_version – MLX version measured on.

• macos_version – macOS version measured on.

• workload – Workload shape this run approximates.

• created_at – ISO-8601 timestamp of when the record was recorded.

• source – Origin of the record ("manual", "veloxquant").

• tags – Free-form labels (e.g. ["metal-kernel"]).

Parameters:

• id (str)

• method (str)

• model_id (str)

• architecture (str)

• context_length (int)

• batch_size (int)

• memory_bytes (int)

• memory_reduction (float | None)

• throughput_tok_s (float)

• latency_ms_per_token (float)

• perplexity_delta (float | None)

• chip (str)

• mlx_version (str)

• macos_version (str)

• workload (WorkloadProfile)

• created_at (str)

• source (str)

• tags (list[str])

id: str

method: str

model_id: str

architecture: str = 'unknown'

context_length: int = 4096

batch_size: int = 1

memory_bytes: int = 0

memory_reduction: float | None = None

throughput_tok_s: float = 0.0

latency_ms_per_token: float = 0.0

perplexity_delta: float | None = None

chip: str = 'unknown'

mlx_version: str = 'unknown'

macos_version: str = 'unknown'

workload: WorkloadProfile

created_at: str

source: str = 'manual'

tags: list[str]

classmethod from_dict(data)

Build a record from its JSON-decoded dict form (inverse of to_dict()).

Return type:

BenchmarkRecord

Parameters:

data (dict[str, Any])

to_dict()

Return a JSON-serializable dict form of this record, including the workload.

Return type:

dict[str, Any]

property savings_percent: float

Memory savings vs the fp16 baseline, as a percentage (0..100).

0.0 when no memory reduction was measured.

match_score(other)

Closeness to another record/query, higher is a better match (0..1).

Compares the variables that change the measured values most: the context, the batch, the chip generation and the MLX version. Perfect match scores 1.0; each-axis difference costs a fraction.

Return type:

float

Parameters:

other (BenchmarkRecord | BenchmarkMatchQuery)

class veloxquant_mlx.benchmarks.benchmark_db.BenchmarkDatabase(storage_dir=None)

Bases: object

Bounded JSON store of KV-cache benchmark records.

All mutations are serialized under a module lock and written atomically (temp file + os.replace), so concurrent writers cannot corrupt the store.

Parameters:

storage_dir (Path | str | None)

load()

Load the index and every record file (skipping corrupt ones).

Return type:

None

save()

Persist the whole store atomically.

Return type:

None

property records: dict[str, BenchmarkRecord]

Shallow copy of all stored records, keyed by record id.

find(record_id)

Look up a single record by id, or None if it isn’t stored.

Return type:

BenchmarkRecord | None

Parameters:

record_id (str)

find_best_match(model, workload, methods=None, *, threshold=0.6, hardware=None)

Best stored record per method, and only when it matches closely enough.

Parameters:

• model (ModelProfile) – Model to match against (architecture + model_id used).

• workload (WorkloadProfile) – Workload to match against.

• methods (list[str] | None) – Candidate methods to look up; all when None.

• threshold (float) – Minimum BenchmarkRecord.match_score(); matches below it are considered not-measured and omitted (avoids handing the planner numbers from an unrelated machine).

• hardware (Any | None) – Optional HardwareProfile; fills the chip / version fields on the query so chip dissimilarity is penalized rather than ignored.

Return type:

dict[str, BenchmarkRecord]

upsert(record)

Insert or replace a record by its id, then persist.

Return type:

BenchmarkRecord

Parameters:

record (BenchmarkRecord)

add(method, model_id, *, context_length, batch_size=1, memory_bytes=0, memory_reduction=None, throughput_tok_s=0.0, latency_ms_per_token=0.0, perplexity_delta=None, chip='unknown', mlx_version='unknown', macos_version='unknown', workload=None, source='manual', tags=None, architecture=None, model=None)

Build, persist, and return a new record (keyed by fingerprint).

Return type:

BenchmarkRecord

Parameters:

• method (str)

• model_id (str)

• context_length (int)

• batch_size (int)

• memory_bytes (int)

• memory_reduction (float | None)

• throughput_tok_s (float)

• latency_ms_per_token (float)

• perplexity_delta (float | None)

• chip (str)

• mlx_version (str)

• macos_version (str)

• workload (WorkloadProfile | None)

• source (str)

• tags (list[str] | None)

• architecture (str | None)

• model (ModelProfile | None)

remove(record_id)

Delete a record by id and persist. Returns False if it wasn’t found.

Return type:

bool

Parameters:

record_id (str)

clear()

Delete all records and persist the now-empty store.

Return type:

None

veloxquant_mlx.benchmarks.benchmark_db.benchmark_fingerprint(method, model_id, context_length, batch_size, chip='unknown')

Stable record id: same method+model+workload shape+chip -> same key.

The chip is part of the key so a measurement taken on another chip does not overwrite the first (the store keeps both and matching scores closeness). An unknown chip hashes exactly as before, so ids of records written without a chip are unchanged (#639).

Return type:

str

Parameters:

• method (str)

• model_id (str)

• context_length (int)

• batch_size (int)

• chip (str)

veloxquant_mlx.benchmarks.benchmark_db.default_benchmark_dir()

~/.cache/veloxquant/benchmarks (respects $XDG_CACHE_HOME).

Return type:

Path