The status endpoint now reports, per account, how many sessions are
bound with traffic inside the routing load window, plus the account's
cooldown deadline. The coderouter CLI uses both to make sticky routing
and rate-limit cooldowns visible. The session-count read is display
only and fails to zero so it can never take the status endpoint down.
Pricing change: a team may connect and route up to 3 provider
accounts (subscriptions) for free. More than 3 requires an active
cmux Pro or Team subscription.
- New services/coderouter/entitlement.ts: free tier checked first
(one indexed count read); the Stripe read runs only for teams over
the limit. Both checks fail closed.
- Session issuance: replaces the flat Pro gate. Over-limit teams
without a subscription get 402 with the count in the message.
- Account add: connecting an account beyond the limit without a
subscription returns 402 before anything is stored. Re-importing
an account the team already has is always allowed, so a broken
account stays repairable on the free tier. The accounts POST
handler is now factory-built for dependency-injected tests.
- Billing-lapse token revocation stays unchanged: after a lapse the
CLI renews and the free tier re-qualifies teams with <= 3 accounts.
- New analytics: entitlement_basis on route_session_issued and a
coderouter_account_limit_reached event, both schema-whitelisted.
- CODEROUTER_HOSTED_PRO_REQUIRED=0 still disables all gating.
Boundary: exactly 3 accounts is free; the 4th needs Pro/Team.
A bound account in state 'refreshing' is healthy; its refresh lease
resolves in seconds. Treat it as usable for sticky reuse, and give a
sticky session a short bounded wait (4 x 500ms) when the refresh lease
is busy, instead of instantly moving the session and discarding its
prompt cache. Non-sticky requests keep the fail-fast move.
If the code deploys before the additive migration is applied, the
session-stickiness statements hit undefined_table (42P01). Treat that
as: no binding found, claim without the session-load ordering term, and
skip the pin. Routing behaves exactly like the pre-change code until
the migration lands, instead of erroring every Codex request. Covered
by a DB behavior test that renames the table away and back.
Port of subrouter PR #228 to the coderouter TypeScript data plane.
Problems in the current routing:
- selectAccountForRequest picks an account per request with three
independent statements (sweep, select, update). Nothing spans the
read-pick-write sequence, so parallel session starts read the same
snapshot and herd onto one account. That account burns quota first,
then every session reroutes to the next account at once.
- No session stickiness exists. Each request rotates to the
least-recently-used account, so the provider prompt cache is
discarded on nearly every turn and the whole prefix is re-billed
as uncached input across all accounts.
Fix:
- New coderouter_session_accounts table pins one agent session (the
Codex CLI session_id header) to one account.
- selectAccountForSession honors a usable binding first (sticky), and
moves a session only when its account is broken, cooling down,
removed, or already attempted in this request.
- New placements claim an account atomically in one statement with
FOR UPDATE SKIP LOCKED, ordered by fewest recently-active bound
sessions, then least-recently-used. Concurrent claims take
different accounts instead of the same snapshot argmax. When every
candidate row is locked, a blocking fallback claim accepts a
collision instead of failing the request.
- selectAccountForRequest (models, opencode) now uses the same atomic
claim, which closes its read-pick-write race too.
Tests: unit tests for the selector and the responses proxy, and
CMUX_DB_TEST-gated behavior tests that prove sequential spread,
concurrent non-herding, stickiness, move-on-cooldown, exclusion,
and last-write-wins binding against real Postgres.
The lane is fleet infrastructure we operate, not contributor guidance for this
codebase, so the prose now lives in cmuxterm-hq at
skills/infra/blacksmith-testbox/ beside macfleet and cloud-vm-ops. See
manaflow-ai/cmuxterm-hq#306.
Everything the box actually executes stays here: both workflows, the five
scripts/blacksmith-*.sh helpers, and the two guards that validate them. Only
tests/test_testbox_doc_blocks.sh follows the skill, because it validates that
prose, and hq gained a workflow so it keeps gating rather than quietly stopping.
CLAUDE.md keeps a pointer rather than dropping the subject, because an agent
starting cmux-tui Rust work reads this file and must still learn not to compile
on the Mac. Worth stating plainly: all nine trial agents found this skill from a
local file in this repo, and a pointer to another checkout is weaker
discoverability, particularly from a plain cmux clone.
Review follow-up: replace the 0/1 ternaries with an exhaustive switch into
DiagnosticConnectionMethod so a future third method becomes a compile error
instead of silently reporting as Tailscale.
Co-Authored-By: Claude Fable 5 <[email protected]>
Review follow-up: inject a fixed date instead of wall-clock Date() so
route-selection behavior in these tests is deterministic.
Co-Authored-By: Claude Fable 5 <[email protected]>
The bundle-isolation rollout (#9183) records iOS builds without the
X-Cmux-App-Namespace header as legacy. Those are the shipped pre-namespace
Beta binaries on the default lane, and they lost the official-namespace
default->{default,nightly} exception: discovery still listed a Nightly Mac,
but issuePairGrant denied it with target_not_pairable, surfacing on the
phone as Authorization failed while relay policy and reachability passed.
Legacy callers on the default lane now get the same default+nightly Mac
reach as official namespaced apps, applied through the single
canIOSBindingUseMac choke point so pair grants and proofed discovery agree.
Non-default legacy lanes keep exact tag matching, and canIOSBindingForgetMac
is split off the alias so the destructive forget path is not broadened.
Co-Authored-By: Claude Fable 5 <[email protected]>
Old Beta builds predate the X-Cmux-App-Namespace header and binding request
proofs, so the broker records them as legacy/default. They currently lose the
official-namespace default->nightly exception and get target_not_pairable
against Nightly Macs. These tests pin the intended compatibility: pairing and
discovery gain the fallback, non-default legacy lanes keep exact matching, and
the destructive forget_mac path is not broadened.
Co-Authored-By: Claude Fable 5 <[email protected]>
Revalidating the demo after the set-difference rewrite passed the build (2m13s
then 0.14s, box stopped, exit 0), but the org check afterwards showed its warmup
run still in_progress, holding a 32 vCPU runner. The skill calls cancelling a
required step and the script did not do it, so the one entry point most people
will run was the one leaking.
It records the run it approves and cancels it from the same EXIT trap that stops
the box, so a Ctrl-C cleans up both.
I added the cargo test line to the skill without running it. A concurrency
trial ran it and it fails on a fresh box: 104 failures, exit 101, all one root
cause. `blacksmith testbox run` gives a shell at umask 0002, so test directories
are created group-writable, and cmux-remote's secure-directory check correctly
rejects an ancestor writable by other users without the sticky bit.
Hosted CI runs at umask 022, so the suite passes there and fails here. The suite
is not umask-independent, which means this lane does not reproduce CI unless the
command says so. With `umask 022` prepended, 3504 tests pass in about 88s.
Builds and clippy are unaffected.
Two agents dispatching nine seconds apart deadlocked. The guard correctly
refused to guess between two identical waiting gates, then told the operator to
"approve yours in the UI". An agent has no UI, and nothing in the REST API binds
a run to a Testbox ID, so the recovery was unusable by the caller the lane is
built for. One trial burned 11 minutes and an extra box getting out of it.
Snapshot the waiting gates before dispatching and take the set difference after.
That is exact where a 120 second window is not, since another operator lands
inside any window you pick. On the rare true tie the instruction is now
deterministic and needs no browser: stop your box, re-snapshot, dispatch again.
Also record the trap the same trial nearly fell into. Correlating a box to a run
by timestamp looks right and is wrong: Blacksmith rewrites a box's CREATED value
as it hydrates, measured moving 05:54:43 to 05:55:03 to 05:58:48 on one box. The
authoritative binding is the RUN URL column, which appears only once the box is
ready.
SKILL.md, benchmark.md, and the demo script now share the one algorithm.
Every trial exercised the evidence path, so the skill only ever showed
cargo build. An agent's real task is usually cargo test or clippy, which use the
same warm target/ and need no stage helper; show them.
Move the trust-boundary reasoning to references/trust-boundary.md and keep the
operative summary inline. An agent about to build needs the rule and the
enforcement, not the full argument. SKILL.md is 11.7 KB, still above the 10 KB
guideline; the remaining bulk is the approval block, and the eighth trial proved
that moving an executable step away from the file that needs it strands a live
box, so it stays.
Ninth trial ran all eight documented blocks literally, in order, each exiting 0,
repairing nothing. Two cosmetic inaccuracies left, both mine.
The EXIT trap printed "Testbox <id> is still running; no stop was authorized"
at the end of a run whose ceremony had already stopped the box, because it
inferred liveness from CONFIRM_TESTBOX_STOP_SHA being unset rather than from
the inventory it had just captured. On a clean run that reads as a failure. It
checks list-at-exit.log now.
And I cited the changed-file overhead gap as "several times larger, 8.1s against
1.2s". It measured 1.48s this run with no change to the lane. The guidance not
to read that gap as sync overhead stands; the magnitude does not, so the text no
longer promises one.
Eighth trial ran the documented blocks literally, as instructed, and the
warmup block stranded a live 32 vCPU box. `grep -c .` exits 1 when the count is
zero, and the plan sets `set -euo pipefail`, so the assignment
`waiting_count="$(... | grep -c .)"` ended the script on the first poll. Zero
waiting runs is the normal state right after warmup returns, so the 150s poll
loop was unreachable dead code, and the exact failure commit 8d3edf5b28 set out
to fix happened again, one round later, in the fix itself. There is no error
text; the script simply stops, leaving a warmed box, an un-approved run, and a
receipt the trap correctly refuses to act on.
Guarded every count with `|| true` in both files.
tests/test_testbox_doc_blocks.sh now treats the plan as executable
documentation: it parses all 21 fenced bash blocks under the shell the plan
mandates, then executes each counting construct against an empty result to prove
it survives its normal first state. Verified red against the unguarded line and
green after. Syntax checking could never have caught this, which is why two
rounds of review missed it.
Two more from the same trial. benchmark.md never cancelled the warmup run, so
following it alone leaves the keepalive holding a runner until the 120 minute
timeout; it now cancels and polls to terminal state. And the setup artifact is
required evidence whose command sat in prose rather than a fenced block, so a
literal run never captured it.
Seventh trial ran smooth end to end, zero unexpected failures, and still found
that benchmark.md's warmup block goes straight from the receipt to
`status --wait`. The approval existed only in the surrounding prose, so anyone
copying the block parks at the environment gate, burns the full 15 minute
timeout, and strands a warmed 32 vCPU box. Prose next to a code block does not
protect someone running the code block. The approval, with the polling guard and
its own attempt transcript, is now in the code, along with the DISPATCH_EPOCH
capture it binds to. Removed the unreachable second warmup_status check while
there.
Also qualified the overhead column. The changed-file gap measured 8.1s against
1.2s for the other stages while all three synced strategy=skip, because that
stage's backup, edit, restore, and re-verify work runs inside the CLI call but
outside wall_seconds. It is real, it is not sync and transport, and
operations.md now says to compare overhead only across first-clean and
incremental-noop.
Every bash block in the plan is now syntax-checked as a unit.
Sixth trial reported the run as not smooth, and it was right. The guard I wrote
in SKILL.md Step 3 queries for waiting runs the instant warmup returns. GitHub
had not surfaced the run yet, so zero matched, and the guard printed "more than
one run waiting; approve yours in the UI" and exited. Zero runs is not
ambiguity, it is "not yet". Under benchmark.md's set -e that abort left a live,
un-approved box on the org inventory with its receipt already written, and the
plan's own trap is designed to refuse to stop it.
The guard now polls for 150s, distinguishes zero from two-or-more, and says
which happened. It also tells you that an abort means you already own a running
box, so stop it before retrying or the next dispatch leaves two boxes and one
receipt.
The trap also lied about why it gave up: it printed "warmup returned no owned
Testbox receipt" when the receipt existed and the real cause was
CONFIRM_TESTBOX_STOP_SHA being unset, which is the normal state. It now reports
the condition that actually held, and names the box you must stop yourself. Its
inventory file was called list-after-warmup-failure.log even on the success
path; it is list-at-exit.log now.
Two smaller ones. Record every approval attempt separately, because a retry that
overwrites the first hides a live unapproved box from the evidence pack. And
warm-before-reading has an exception for evidence runs, whose preflight and
receipt must exist before warmup; three agents in a row hit that ordering.
Finally, changed-file rebuilds cmux-remote as well as cmux-tui, so its ~8s is a
small-edit figure, not a single-crate floor.
Lawrence Chen