veloxquant_mlx.cli.serve module

veloxquant serve — OpenAI-compatible server with a VeloxQuant KV cache.

Thin launcher, per option (a) of issue #27: load the model, wire our cache in via patch_model_kv_cache, hand off to mlx_lm.server. The serving contract itself is inherited — 35 of 40 caches subclass mlx_lm’s KVCache and get update_and_fetch / trim / nbytes for free.

What is not thin is the honesty layer, and it is the reason this file is longer than the ten lines #27 predicted:

• An unsupported method must fail before the model loads, not with an AttributeError on the first HTTP request.

• Byte counters are accounting-only. The cache stores dequantized fp16, so nbytes over-reports and no memory is actually saved. A server is exactly where users assume otherwise, so the warning is unconditional.

• There is no fp16 fallback, silent or otherwise. A method that cannot serve stops the process.

Model-identity keying for prefix-cache reuse (issue #310): mlx_lm.server keys its prompt-prefix cache on model_key = (model_path, adapter_path, draft_model_path) – a plain path tuple, not a content hash. Swapping weights at the same path between requests could serve a stale prefix cache. This is pre-existing mlx_lm behavior, not something this file introduces or fixes – doing so would mean forking ModelProvider/APIHandler internals, against this file’s own “thin launcher” design above. Track upstream if this needs fixing.

veloxquant_mlx.cli.serve.READY_PREFIX = 'VELOXQUANT_READY '

stdout handshake for the control panel. Printed once the model is loaded and the cache is wired, immediately before mlx_lm.server binds the port. Lets the panel distinguish “starting” from “running” without racing a health poll against a slow model load.

veloxquant_mlx.cli.serve.build_parser()

Build the argparse parser for veloxquant serve (model, method, bits, host/port, –set).

Return type:

ArgumentParser

veloxquant_mlx.cli.serve.parse_overrides(pairs, method=None)

Turn FIELD=VALUE strings into typed KVCacheConfig kwargs.

Types come from the dataclass via the registry, so a value that the config would reject is refused here — with the field name in the message — instead of surfacing as an opaque failure deep in cache construction.

When method is given, a field with no effect for that method (e.g. --set kivi_group_size=64 while --method h2o) prints a warning instead of silently doing nothing (issue #345) — a warning, not a hard error, since _CONFIG_FIELDS does not yet curate every method and a false positive there must not block a legitimate override.

Return type:

dict[str, Any]

Parameters:

• pairs (list[str])

• method (str | None)

veloxquant_mlx.cli.serve.validate_method(method)

Reject unservable methods before the expensive model load.

Deliberately fatal. #27’s crash tier includes turboquant_prod — the library’s default — so silently substituting a working method here would mean users think they benchmarked one thing and served another.

Return type:

None

Parameters:

method (str)

veloxquant_mlx.cli.serve.build_config(args)

Build a KVCacheConfig from –method/–bits/–seed plus parsed –set overrides.

Return type:

Any

Parameters:

args (Namespace)

veloxquant_mlx.cli.serve.attach_cache(model, config)

Point model.make_cache at a freshly built VeloxQuant cache list.

Not patch_model_kv_cache: that helper builds one list eagerly and pins it, which suits generate() but would make every concurrent request share one cache. Building per call gives each request its own.

Must run on the thread that will generate — see _Provider._load.

Returns (n_layers, is_batchable). is_batchable is derived from this same probe list rather than by calling make_prompt_cache(model) again after the patch below is applied: that second call would invoke the now-self-referential model.make_cache, which recurses into KVCacheBuilder.for_model until Python’s recursion limit is hit (VeloxQuant-MLX#506; confirmed 333 full re-executions, masked by a broad except Exception further down the call stack, costing 30-70s per real server cold start). Reusing the probe list is strictly cheaper than the old second call, not just recursion-free.

Return type:

tuple[int, bool]

Parameters:

• model (Any)

• config (Any)

veloxquant_mlx.cli.serve.emit_ready(args, n_caches)

Print the machine-readable handshake the control panel waits for.

Return type:

None

Parameters:

• args (Namespace)

• n_caches (int)

veloxquant_mlx.cli.serve.run_server(args)

Hand off to mlx_lm’s server, wiring our cache in as it loads the model.

The model is loaded by mlx_lm itself, inside its generation thread, rather than up front on the main thread. MLX arrays are bound to the thread that created them, so a main-thread model produces “There is no Stream(gpu, N) in current thread.” the first time the generation thread evaluates cache state. Upstream sidesteps this by calling load_default() from within _generate; hooking _load puts our patch on that same thread.

Return type:

None

Parameters:

args (Namespace)

veloxquant_mlx.cli.serve.main(argv=None)

Parse CLI args and launch the OpenAI-compatible server with a VeloxQuant KV cache.

Validates –method fails fast before the model loads, prints the unconditional accounting-only warning, and hands off to mlx_lm.server (via run_server) with the cache wired in through attach_cache.

Return type:

None

Parameters:

argv (list[str] | None)