MTPLX 2.6.0 — concurrency

Until now, speculative decoding was a single-user feature: the moment two requests hit the daemon at once, everyone fell back to plain autoregressive batching and lost the MTP speedup. 2.6.0 removes that trade-off. This release also brings embedding and reranking endpoints, LiquidAI LFM2 support, and a real correctness fix to temperature-0 decoding.

Concurrent speculative decoding (--scheduler-mode mtp_batch)

The new scheduler serves independent requests through fixed-width MTP cohorts. Each row owns its own state and sampling decisions, drafts are verified in one batched target forward, and rows join and leave mid-flight without disturbing their neighbours. Two cohort widths (three-wide and eight-wide) install side by side and the scheduler seals each cohort at the narrowest width that fits, so two concurrent agents don't pay for eight lanes of padding.

Measured on Qwen3.6-35B-A3B on an M5 Max: the three-wide lane holds 1.7-1.8x the per-request decode of the padded eight-lane shape, and against the previous production ar_batch route the same concurrent agent workloads decode at 1.6-2.25x per lane, sampled at the model's shipped settings.

Honesty controls ship with it: --mtp-batch-numerics picks between throughput, balanced, and b1-exact profiles with documented trade-offs and an install-time self-check, and per-request stats report each row's own truth (its own accepted-depth histogram, cohort width, and restore provenance) instead of cohort averages.

The session bank composes with the cohorts: a request whose prefix is banked restores it at cohort admission, prefills only its uncovered suffix, and commits its own prompt boundary before the merge — agent fleets with a shared system prompt keep warm time-to-first-token under concurrency. The plain ar_batch lane learned the same trick.

This work is by David Tai (@davidtai): the scheduler contract, the row-owned decode, the cohort serving, the numerics profiles, and the docs. The width-3 bucket, session-bank composite, and live QA came out of the joint hardening passes on top.

Embeddings and reranking (/v1/embeddings, /v1/rerank)

Contributed by @Cyb3rb1ade (PR #212). The daemon can now serve embedding and reranker models beside chat, so a retrieval-backed setup doesn't need a second inference server. OpenAI-shape embeddings (including the dimensions Matryoshka truncation), Cohere/Jina-shape rerank, opt-in per-model flags, lazy loading, an LRU resident cap, and idle release under memory pressure. /v1/models stays chat-only by default so model pickers never offer an embedder as a chat target; retrieval-only ids answer chat requests with a clear 400. Checkpoints that ship their own Python code are refused unless you explicitly pass --retrieval-trust-remote-code.

LiquidAI LFM2 / LFM2.5

By David Tai (@davidtai). The LFM2 family serves natively with a bit-exact ShortConv decode fast-path and a verified think/tool grammar (parser stamp, native tool prompt, pythonic streaming dialect). IQuest-Coder checkpoints serve target-only AR through the same registry honesty: recognized, served without MTP claims, refused cleanly when the quantization can't execute.

Temperature-0 output is token-identical again

The speculative lane's cold prefill fed the whole prompt through the model in one window while plain decoding splits it into body plus a final single-token step. The two shapes round differently in the last bit, so the speculative lane started from a cache one ulp apart from the plain lane's — enough to flip greedy argmax at a near-tie and break the "temperature 0 matches plain decoding" contract. Every cold-prefill path now partitions the prompt identically. The Optimized Speed V2 artifact, which surfaced the flip, passes its greedy exactness gate at every depth again.

Fixes

• Prefix restores no longer corrupt the session bank (#247): restores install fresh zero-copy views, so an interleaved near-prefix request can never rewrite a neighbour's banked span. From the reporter's reproducer, which ships as a regression test.
• Streaming tool calls no longer duplicate argument values into delta.content (#249).
• Solo requests carrying presence/frequency penalties answer on the composite scheduler instead of returning HTTP 500.
• mtplx serve --no-auth actually parses (#235 follow-through) — 2.5.4 promised it, the public CLI rejected it.
• qwen3_5_mtp checkpoints validate and serve again — the MTP surface now attaches at the TextModel level. Cherry-picked from PR #242 by @davidtai with its regression test.
• Artifacts launch at their measured depth again: the typed runtime contract silently dropped measured-depth maps, so 35B launched at its D3 ceiling — a measured ~22% decode loss against its fastest depth.
• --reasoning-parser is authoritative; the backend codec no longer silently overrides an operator-typed parser.
• Metal buffer-object leak in long decodes: mlx-lm's ArraysCache regrew buffer objects on every advance; the fix is vendored in-tree so pip installs against stock mlx-lm 0.31.x get it too.
• Missing MTP heads degrade to target-only AR serving with the reason surfaced, instead of refusing the checkpoint. auto_map checkpoints are refused with the policy stated plainly; unrunnable quantizations are refused with the offending bit-width named.
• OpenAI dimensions honored on /v1/embeddings; out-of-range values are a clear 400.
• AR batch hardening: cache-removal errors fail closed, completed streams no longer starve behind running neighbours, and the vendored Metal shader cache is keyed by MLX ABI so an MLX upgrade can't serve stale kernels.

QA (this release)

• Full pytest battery green at the release tip; 567 Swift app tests green.
• Greedy exactness gate for Optimized Speed V2: 3/3 depths token-exact (was 0/3 on 2.5.4).
• Four-arm performance sweep against the shipped 2.5.2, 2.5.3, and 2.5.4 wheels (same harness bytes, fans verified at max, die-temp gated, candidate interleaved with baseline): single-request decode flat-to-faster than 2.5.4 (interleaved means 74.8 vs 69.2 tok/s on the 27B artifact), cold TTFT flat (6.25s vs 6.26s on a 4.8k-token prompt), warm TTFT flat at the 2.5.4 session-bank floor (0.14s vs 0.15s). A serial-lane sampling regression introduced mid-cycle was caught by this sweep and fixed before release.
• Live QA on both product surfaces at the release tip: mtplx serve from a clean wheel install (streamed think and content, 64-70 tok/s) and the macOS app driven end-to-end (chat round trip, 52.5 tok/s reported by the app, clean stop and quit).

Credits

• David Tai (@davidtai) — concurrent MTP serving stack, LFM2 support, qwen3_5_mtp fix (PR #242). 26 commits in this release.
• @Cyb3rb1ade — embeddings + rerank endpoints (PR #212, 17 commits).