veloxquant_mlx.planning.workload module

Workload description for the auto-selection pipeline.

A WorkloadProfile captures everything about how the model will be used that changes which KV-cache strategy fits (context length, generation length, batch size/parallelism, raw compute vs. response-time goal). It is pure data — no methods, just fields — so it can be round-tripped through JSON, the benchmark database, and the CLI without impedance mismatch.

class veloxquant_mlx.planning.workload.WorkloadProfile(context_length=4096, generation_length=512, batch_size=1, num_concurrent_requests=1, objective='balanced', max_latency_ms=None)

Bases: object

Normalized description of a serving workload.

Variables:

• context_length – Maximum prompt context the served cache must hold.

• generation_length – Expected number of decode tokens per request.

• batch_size – Parallel requests sharing the KV cache.

• num_concurrent_requests – Requests served concurrently (drives how many caches the memory budget must cover).

• objective – One of the WorkloadObjective names; the planner uses this to weight the memory/latency/throughput/quality axes.

• max_latency_ms – Hard latency ceiling, when the caller knows one (used to exclude strategies whose analytical proxy exceeds it).

Parameters:

• context_length (int)

• generation_length (int)

• batch_size (int)

• num_concurrent_requests (int)

• objective (str)

• max_latency_ms (float | None)

context_length: int = 4096

generation_length: int = 512

batch_size: int = 1

num_concurrent_requests: int = 1

objective: str = 'balanced'

max_latency_ms: float | None = None

to_dict()

Serialize all fields to a plain dict.

Return type:

dict[str, Any]

property total_tokens_per_request: int

Context + generation, the cache lifetime of one request.

property effective_batch: int

Attention batch the memory estimator must budget for.

class veloxquant_mlx.planning.workload.WorkloadObjective

Bases: object

Identifier constants for the planner’s supported objectives.

Kept as a plain object (not an Enum) so downstream code can pass arbitrary objective names; the planner’s DEFAULT_OBJECTIVE_WEIGHTS only assigns weights to the canonical ones below and falls back to balanced for anything else.

MEMORY = 'memory'

LATENCY = 'latency'

THROUGHPUT = 'throughput'

QUALITY = 'quality'

BALANCED = 'balanced'

veloxquant_mlx.planning.workload.workload_from_dict(data)

Rehydrate a WorkloadProfile from to_dict output.

Missing keys take their dataclass defaults; extra keys are ignored, so the schema can grow without breaking loading of older records.

Return type:

WorkloadProfile

Parameters:

data (dict[str, Any])