Duke/cmux
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: move the Testbox skill to cmuxterm-hq

Browse Source 
The lane is fleet infrastructure we operate, not contributor guidance for this
codebase, so the prose now lives in cmuxterm-hq at
skills/infra/blacksmith-testbox/ beside macfleet and cloud-vm-ops. See
manaflow-ai/cmuxterm-hq#306.

Everything the box actually executes stays here: both workflows, the five
scripts/blacksmith-*.sh helpers, and the two guards that validate them. Only
tests/test_testbox_doc_blocks.sh follows the skill, because it validates that
prose, and hq gained a workflow so it keeps gating rather than quietly stopping.

CLAUDE.md keeps a pointer rather than dropping the subject, because an agent
starting cmux-tui Rust work reads this file and must still learn not to compile
on the Mac. Worth stating plainly: all nine trial agents found this skill from a
local file in this repo, and a pointer to another checkout is weaker
discoverability, particularly from a plain cmux clone.
This commit is contained in:
Lawrence Chen
2026-08-18 20:40:08 -07:00
parent 540bf2b9e4
commit bc0dcaa8b9
7 changed files with 4 additions and 1211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/testbox-broker-guard.yml
-3
View File
@@ -43,9 +43,6 @@ jobs:
- name: Validate receipt-bound cleanup ownership check
run: ./tests/test_testbox_cleanup_receipt_ref.sh
- name: Validate the documented shell blocks
run: ./tests/test_testbox_doc_blocks.sh
- name: Lint the Testbox lane helpers
run: shellcheck scripts/blacksmith-bounded-command.sh scripts/blacksmith-cmux-tui-testbox-stage.sh scripts/blacksmith-testbox-cleanup.sh scripts/blacksmith-testbox-keepalive.sh
CLAUDE.md
+4 -1
View File
@@ -131,4 +131,7 @@ Detailed contributor rules live in `skills/`. Use the task-specific skill before
- `cmux-shared-behavior`: shared action paths for multi-entrypoint behavior and optimistic updates.
- `cmux-ghostty`: Ghostty submodule and GhosttyKit workflow.
- `cmux-release`: release, version bump, changelog, pretag guard, release assets.
- `blacksmith-testbox`: warm your own Linux Testbox before any cmux-tui Rust or Zig build, and never compile cmux-tui on the Mac.
- Blacksmith Testbox (remote Linux builds for cmux-tui): warm your own box before any cmux-tui Rust or Zig
build, and never compile cmux-tui on the Mac. The skill lives in cmuxterm-hq at
`skills/infra/blacksmith-testbox/SKILL.md`; the workflows, `scripts/blacksmith-*.sh`, and the
`tests/test_testbox_*` guards stay here. Quickest path: `./scripts/blacksmith-testbox-demo.sh`.
skills/blacksmith-testbox/SKILL.md
-282
View File
@@ -1,282 +0,0 @@
---
name: blacksmith-testbox
description: >
Warm and drive a Blacksmith Testbox for cmux-tui Rust and Zig work on Linux.
Use whenever a task will compile cmux-tui (cargo build, cargo test, clippy,
startup benchmarks, Ghostty Zig deps) or when the user says testbox, warm a
box, blacksmith, remote build box, or benchmark cmux-tui. Never run cargo,
rustc, or zig build on the local Mac.
---
# cmux-tui Blacksmith Testbox
A Testbox is a persistent 32 vCPU Linux VM that keeps its disk between
commands. CI warms it once, and every later build reuses the Cargo registry,
the Zig cache, and `target/`. A clean cmux-tui build costs about 130 s on a warm
box, a no-op about 0.2 s, and a one-file change about 8 s.
## Warm your own box first, before you read code
Warmup takes about four minutes of wall clock you cannot compress. Starting it
at the end of your task means waiting; starting it in your first minute means it
is ready by the time you know what to build. So:
1. If your task might compile cmux-tui, dispatch the warmup **before** you open
a single source file, then read code while it hydrates. The exception is an
evidence run: `benchmark.md` requires its preflight, `OUT_ROOT`, and receipt
to exist before warmup, so read that plan first and accept the delay.
2. Warm **your own** box. One box belongs to one worktree and one agent, because
`blacksmith testbox run` synchronizes your worktree onto it with
`rsync --delete`. Two agents on one ID overwrite each other's source and
corrupt each other's timings.
3. Never adopt a box you find in `blacksmith testbox list --all`. A box you did
not warm is someone else's working state.
4. Stop your box the moment you are done, and always pass `--idle-timeout` so a
crashed agent cannot leak a running VM.
Skip the lane entirely for Swift, Xcode, XCUITest, GUI, and app-host work. That
is macOS, and it belongs in the hosted macOS workflows.
Run `./scripts/blacksmith-testbox-demo.sh` once to watch the whole lane work:
it warms a box, pins it, builds cmux-tui twice to show what the persistent disk
buys, prints every remote command before running it, and stops the box on the
way out. `--stages` runs the three measured stages instead.
## Prerequisites
```bash
blacksmith auth whoami # confirms the manaflow-ai org; prints no secret
blacksmith --version # record this with any evidence
```
Use the organization-approved pinned CLI. Never install it with
`curl ... | sh`. If the pinned artifact is unavailable, stop and ask the tooling
owner rather than substituting a version.
Run every command from the root of an isolated cmux worktree, never from
`repo/`. The directory you stand in is the directory that gets synchronized.
## Workflow
### Step 1: commit, push, and record identity
The commit you benchmark must be pushed. `blacksmith testbox run` synchronizes
file contents, not history: it makes one opportunistic `git fetch` of your
commit, falls back to copying changed files, and skips even that once the file
fingerprints match. A local-only commit fails on the box with
`upload-pack: not our ref`.
```bash
git submodule update --init ghostty
git push origin "$(git symbolic-ref --short HEAD)"
SOURCE_SHA="$(git rev-parse HEAD)"
GHOSTTY_SHA="$(git rev-parse HEAD:ghostty)"
[[ -z "$(git status --porcelain=v1 --untracked-files=normal)" ]] || exit 1
```
### Step 2: warm the box
```bash
blacksmith testbox warmup .github/workflows/cmux-tui-testbox-warmup.yml \
--ref main --job cmux-tui-rust --idle-timeout 30 # minutes, not seconds
TBX=tbx_... # the ID the command prints; assign it once and never retype it
```
`--ref main` is the trust boundary, not a preference. See the trust section
below. `--idle-timeout` is in minutes.
### Step 3: approve the deployment
`warmup` returns the ID immediately and does not block; the run then parks at
the `blacksmith-testbox-trusted` environment gate before its first step.
Approve it on the run page, or from the shell.
Every run in this lane has the same title and the same `main` head branch, so
**never approve `workflow_runs[0]`**. Two agents warm boxes minutes apart, and
approving the newest waiting run hands a stranger's deployment its gate. Bind
the approval to a run that appeared after your own dispatch, and refuse to guess
when more than one is waiting:
Identify your run by set difference against a snapshot taken before you
dispatch. A time window is not enough: another agent dispatching seconds after
you lands inside any window, and nothing in the REST API binds a run to a
Testbox ID. Never correlate a box to a run by timestamp either, because
Blacksmith rewrites a box's `CREATED` value as it hydrates, so the pairing that
looks obvious is wrong for any box past `queued`.
```bash
# Snapshot every gate already waiting in this lane BEFORE dispatching. Set
# difference against this is exact; a time window is not, because a second
# operator dispatching seconds after you lands inside any window you pick.
lane_runs_url="repos/manaflow-ai/cmux/actions/workflows/cmux-tui-testbox-warmup.yml/runs?event=workflow_dispatch&status=waiting"
waiting_before="$(mktemp)"
gh api "$lane_runs_url" --jq '.workflow_runs[].id' | sort >"$waiting_before"
# ... run `blacksmith testbox warmup` here, then:
# Your run is the one that appeared since the snapshot. GitHub does not surface
# it instantly, so poll; zero new runs means "not yet".
waiting_now="$(mktemp)"
approval_run=""
for attempt in $(seq 1 30); do
gh api "$lane_runs_url" --jq '.workflow_runs[].id' | sort >"$waiting_now"
new_runs="$(comm -13 "$waiting_before" "$waiting_now")"
new_count="$(printf '%s' "$new_runs" | grep -c . || true)"
if (( new_count == 1 )); then
approval_run="$new_runs"
break
fi
if (( new_count > 1 )); then
# Two operators dispatched between polls. Nothing in the REST API binds a
# run to a Testbox ID, so do not guess and do not correlate by timestamp:
# Blacksmith rewrites a box's CREATED value as it hydrates. Stop your box,
# re-snapshot, and dispatch again; the next set difference is unambiguous.
echo "$new_count runs appeared at once; stop your box ($TBX), re-snapshot, and re-dispatch" >&2
exit 1
fi
sleep 5
done
if [[ -z "$approval_run" ]]; then
echo "no run appeared within 150s; Testbox $TBX is running and you own it" >&2
exit 1
fi
approval_env="$(gh api "repos/manaflow-ai/cmux/actions/runs/$approval_run/pending_deployments" --jq '.[0].environment.id')"
gh api -X POST "repos/manaflow-ai/cmux/actions/runs/$approval_run/pending_deployments" \
--input - <<< "{\"environment_ids\":[$approval_env],\"state\":\"approved\",\"comment\":\"benchmark warmup\"}"
printf 'approved run: %s\n' "$approval_run"
```
If this aborts, a warmed box is already running and you own it. Stop it, take a
fresh snapshot, and dispatch again; the next set difference is unambiguous. Do
not wait for a tie to break on its own, because the other operator is probably
stuck at the same guard.
`gh` has no native approve verb for deployments, so this uses REST. Self-approval
is permitted on this environment. `scripts/blacksmith-testbox-demo.sh` implements
exactly this guard if you would rather not hand-roll it.
### Step 4: wait for hydration, then read code
```bash
blacksmith testbox status --id "$TBX" --wait --wait-timeout 15m
```
This blocks while CI installs the pinned Zig and Rust and fetches Cargo and Zig
dependencies. Do your reading now.
### Step 5: pin the box to your commit
The box is an exact checkout of `main`, because that is what CI hydrated. Make
it an exact checkout of your revision. Once per box, before the first build:
```bash
blacksmith testbox run --id "$TBX" \
"set -euo pipefail; git fetch --no-tags origin $SOURCE_SHA; git reset --hard $SOURCE_SHA; git submodule update --init --depth 1 ghostty"
```
### Step 6: build or benchmark
Any build command goes inside `blacksmith testbox run`:
```bash
blacksmith testbox run --id "$TBX" "cd cmux-tui && cargo build -p cmux-tui --locked"
```
Tests and lints work the same way; only the quoted command changes:
```bash
blacksmith testbox run --id "$TBX" "cd cmux-tui && umask 022 && cargo test --locked"
blacksmith testbox run --id "$TBX" "cd cmux-tui && cargo clippy --locked --all-targets -- -D warnings"
blacksmith testbox run --id "$TBX" "cd cmux-tui && umask 022 && cargo test -p cmux-tui-core --locked some_test_name"
```
**`cargo test` needs `umask 022`.** `blacksmith testbox run` gives you a shell
at `umask 0002`, so test directories are created group-writable, and
`cmux-remote`'s secure-directory check rejects any ancestor writable by other
users without the sticky bit. Without the umask a fresh box fails 104 tests with
`PermissionDenied ... has an ancestor writable by other users without
sticky-directory protection`. Hosted CI runs at `umask 022`, so the suite passes
there and fails here; the suite is not umask-independent. With it, 3504 tests
pass in about 88 s. Builds and clippy are unaffected.
Each of these reuses the same warm `target/`, so a second invocation compiles
only what changed. Nothing here needs the stage helper; that is only for
measured timings.
For measured timings use the stage helper, which verifies VM identity, source
identity, and clean status before and after each build:
```bash
blacksmith testbox run --id "$TBX" \
"CMUX_TESTBOX_REMOTE=1 CMUX_TESTBOX_ID=$TBX ./scripts/blacksmith-cmux-tui-testbox-stage.sh first-clean $SOURCE_SHA $GHOSTTY_SHA"
```
Stages are `first-clean` (wipes `target/`, dependencies stay warm),
`incremental-noop` (rebuild with nothing changed), and `changed-file` (appends a
comment to `cmux-tui/crates/cmux-tui/src/main.rs`, builds, restores the bytes).
`changed-file` rebuilds two crates, `cmux-remote` and `cmux-tui`, so read its
~8 s as a small-edit figure, not as a single-crate floor.
`CMUX_TESTBOX_REMOTE=1` guards against an accidental local launch; it is not
authentication. Add `--debug` to `run` to see the sync strategy.
### Step 7: download, then stop
Download after each stage, because the next sync can delete remote output.
```bash
blacksmith testbox download --id "$TBX" testbox-benchmark/first-clean.json ./first-clean.json
blacksmith testbox stop --id "$TBX"
blacksmith testbox list --all # the box is gone from Blacksmith's inventory
gh run list --repo manaflow-ai/cmux --workflow cmux-tui-testbox-warmup.yml --limit 3
```
Check both. `list --all` only shows boxes, and an empty list is **not** proof
that nothing is burning: the warmup run's keepalive step keeps holding a 32 vCPU
runner for a while after the box is gone, and it has a 120 minute job timeout. If
your run is still `in_progress` a couple of minutes after the stop, end it:
```bash
gh run cancel <run-id> --repo manaflow-ai/cmux
```
Cancelling is required, not a fallback. Stopping the box does **not** reliably
end its run: measured runs sat `in_progress` for four minutes afterwards, and
`gh run cancel` itself takes about five minutes to land, so `in_progress` right
after either action is expected. Poll until the run reports `completed`. A
`cancelled` conclusion is the healthy end state here; a run that never reached
`Testbox ready` is the real failure.
That bare `stop` is the right cleanup for ordinary build work. The receipt-bound
ceremony in `benchmark.md` applies only to benchmark evidence someone else will
rely on, where the point is proving the box you destroyed is the one your
receipt describes. Never fabricate a receipt to satisfy that guard.
Wrap any of these in `./scripts/blacksmith-bounded-command.sh <seconds> <cmd>`
so a hung sync cannot stall a session. `benchmark.md` is the full evidence plan;
`references/operations.md` covers stage orchestration, cleanup, and the two
clocks.
## Trust boundary, in short
`begin-testbox` writes an auth token into the CI job, and `warmup` resolves the
workflow definition and the hydrated source from the same `--ref`. Warming a
candidate branch would therefore run that branch's copy of the workflow beside
the token, and the branch could delete its own guards. So the lane hydrates
`main` only, nothing from the repository runs before the token, and your
revision arrives afterwards through `blacksmith testbox run`.
`tests/test_ci_testbox_broker_guard.py` enforces that shape on every pull
request. Never edit the workflow to work around a missing control; if the
`blacksmith-testbox-trusted` environment drifts from required reviewers, no
secrets, no admin bypass, and a branch rule of exactly `main`, disable the lane
and stop. Full reasoning: `references/trust-boundary.md`.
## Rules
- MUST NOT run `cargo`, `rustc`, `rustup`, or `zig build` on the local Mac, including as a fallback when the box is unavailable.
- MUST run every Blacksmith command from the intended worktree root. `rsync --delete` can remove remote files that are absent locally.
- MUST push the benchmarked commit and pin the box to it. An unpushed commit silently leaves the box on `main` with your files written over it.
- MUST NOT reuse another agent's Testbox ID, and MUST NOT run concurrent `run` or `download` commands against one ID from separate clients.
- MUST stop the box and confirm with `list --all` when finished.
- MUST NOT print or download `/tmp/.testbox/auth_token`.
- MUST NOT dispatch the warmup for a pull request, a fork, or any ref other than `main`.
- If the toolchain differs from what CI hydrated, the stage helper exits 66. Rebase onto `main` rather than reporting a cold-cache timing.
skills/blacksmith-testbox/benchmark.md
-602
View File
@@ -1,602 +0,0 @@
# cmux-tui Testbox timing plan
Run this plan from the root of the isolated cmux worktree. It never invokes a
Rust tool on the local Mac. Every `cargo`, `rustc`, and `zig` command below is
inside a quoted command passed to `blacksmith testbox run`, or inside the
setup-only GitHub job on the remote Linux runner.
## Fixed lane contract
| Item | Value |
| --- | --- |
| Workflow | `.github/workflows/cmux-tui-testbox-warmup.yml` |
| Job | `cmux-tui-rust` |
| Runner | `blacksmith-32vcpu-ubuntu-2404` |
| Protected environment | `blacksmith-testbox-trusted` |
| Rust source of truth | `cmux-tui/rust-toolchain.toml`, via `./.github/actions/setup-cmux-tui-rust` |
| Remote build helper | `scripts/blacksmith-cmux-tui-testbox-stage.sh` |
| Cleanup helper | `scripts/blacksmith-testbox-cleanup.sh` |
| Remote output | `testbox-benchmark/` |
A repository administrator must configure the protected environment with
required reviewers, no secrets, administrator bypass disabled, and a deployment
branch rule of exactly `main` before this plan is usable. The lane needs no
environment variables: the workflow refuses any ref except `refs/heads/main`,
so `main` alone decides what code runs in the token-bearing job. Verify that
configuration before each run, and stop instead of treating the environment
name as a guard if it drifts.
`begin-testbox` exposes its auth token to commands in the Testbox, so
`contents: read` is not a trust boundary and the token is not sandboxed. The
repository does not currently pin a checksum-verified Blacksmith CLI artifact;
that is a trusted-lane operational limitation. Use only the organization-
approved CLI, record `blacksmith --version`, and stop rather than silently
substituting a version.
The warmup job only checks out `main`, initializes `ghostty`,
installs Linux headers/tools, installs the pinned Zig and Rust toolchains,
fetches Cargo and Zig dependencies, and records JSON identity. `zig build
--fetch` is the only build-system operation in warmup, and it exits before
compiling. Rust builds happen only in the three explicit benchmark runs.
The current Blacksmith catalog reports the requested x64 label as 32 vCPU and
121.6 GB, while the ARM label with the same vCPU count reports 96 GB. Keep the
requested `blacksmith-32vcpu-ubuntu-2404` label unless repository Linux
constraints make x64 impossible, and record the catalog result with the run.
## Exact benchmarked source and Ghostty identity
Warmup hydrates `main`. This section pins the separate commit you benchmark,
which `blacksmith testbox run` synchronizes onto the warm box. A raw commit SHA
is not a supported warmup ref (HTTP 422, `No ref found`), and this lane accepts
only `main` regardless. Carry the benchmarked SHA as an assertion:
```bash
set -euo pipefail
git submodule update --init ghostty
cd "$(git rev-parse --show-toplevel)"
SOURCE_REF="$(git symbolic-ref --short HEAD)"
if [[ ! "$SOURCE_REF" =~ ^[A-Za-z0-9._/-]+$ || "$SOURCE_REF" == *..* || "$SOURCE_REF" == */ || "$SOURCE_REF" == *//* ]]; then
echo "HEAD must name a supported pushed branch ref" >&2
exit 1
fi
SOURCE_SHA="$(git rev-parse HEAD)"
SOURCE_TREE_SHA="$(git rev-parse 'HEAD^{tree}')"
ghostty_entry="$(git ls-tree HEAD ghostty)"
[[ "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ]] || {
echo "HEAD:ghostty is not a gitlink" >&2
exit 1
}
GHOSTTY_SHA="$(git rev-parse HEAD:ghostty)"
[[ -n "$SOURCE_REF" ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$GHOSTTY_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$(git -C ghostty rev-parse HEAD)" == "$GHOSTTY_SHA" ]]
[[ -z "$(git status --porcelain=v1 --untracked-files=normal)" ]]
[[ -z "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]]
remote_sha="$(git ls-remote --exit-code origin "refs/heads/$SOURCE_REF" | awk 'NR == 1 { print $1 }')"
[[ "$remote_sha" == "$SOURCE_SHA" ]] || {
echo "push the exact clean branch head before warming Testbox" >&2
exit 1
}
EVIDENCE_ROOT="$PWD/.cmux-scratch"
# Every evidence directory is blacksmith-testbox-<sha>-<suffix>, so test the
# glob, not the bare name. The bare name never matched, so a second run of the
# same SHA from the same PID collided instead of getting a timestamp.
if compgen -G "$EVIDENCE_ROOT/blacksmith-testbox-$SOURCE_SHA-*" >/dev/null; then
RUN_SUFFIX="$(date -u +%Y%m%dT%H%M%SZ)-$$"
else
RUN_SUFFIX="initial-$$"
fi
OUT_ROOT="$EVIDENCE_ROOT/blacksmith-testbox-$SOURCE_SHA-$RUN_SUFFIX"
if [[ -e "$OUT_ROOT" ]]; then
echo "evidence directory already exists; choose a new run path: $OUT_ROOT" >&2
exit 1
fi
mkdir -p "$OUT_ROOT/raw"
python3 - "$SOURCE_REF" "$SOURCE_SHA" "$SOURCE_TREE_SHA" "$GHOSTTY_SHA" > "$OUT_ROOT/source.json" <<'PY'
import json
import sys
ref, sha, tree, ghostty = sys.argv[1:]
print(json.dumps({
"source_ref": ref,
"source_sha": sha,
"source_tree_sha": tree,
"ghostty_gitlink_sha": ghostty,
}, indent=2, sort_keys=True))
PY
assert_source_unchanged() {
local current_ref current_sha current_tree current_ghostty remote_sha ghostty_entry
current_ref="$(git symbolic-ref --short HEAD)"
current_sha="$(git rev-parse HEAD)"
current_tree="$(git rev-parse 'HEAD^{tree}')"
ghostty_entry="$(git ls-tree HEAD ghostty)"
current_ghostty="$(git rev-parse HEAD:ghostty)"
remote_sha="$(git ls-remote --exit-code origin "refs/heads/$SOURCE_REF" | awk 'NR == 1 { print $1 }')"
if [[ "$current_ref" != "$SOURCE_REF" || "$current_sha" != "$SOURCE_SHA" ||
"$current_tree" != "$SOURCE_TREE_SHA" || "$current_ghostty" != "$GHOSTTY_SHA" ||
! "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ||
"$remote_sha" != "$SOURCE_SHA" ||
-n "$(git status --porcelain=v1 --untracked-files=normal)" ||
-n "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]]; then
echo "source branch, tree, Ghostty gitlink, remote head, or clean status changed" >&2
return 1
fi
}
assert_source_unchanged
```
`assert_source_unchanged` runs before every stage below. If the branch moved,
the worktree became dirty, or the Ghostty pointer changed, stop the box and
start a new evidence directory. Do not silently substitute the new SHA.
## Warmup and setup identity
Warm from `main`. The lane refuses every other ref, and a raw SHA is not a
supported warmup ref. Your benchmarked branch never appears here; it reaches the
box later, through the pin step.
**Then approve the deployment gate, before you wait.** The run parks at the
`blacksmith-testbox-trusted` environment gate before its first step, so the
`status --wait` below simply times out after 15 minutes if nothing approves it.
Use the `DISPATCH_EPOCH`-guarded approval in `SKILL.md` Step 3, which binds to a
run created after your own dispatch and refuses when more than one is waiting.
Never approve `workflow_runs[0]`: every run in this lane shares a title and a
`main` head branch, so the newest waiting run may belong to another operator.
Write every approval attempt to its own `$OUT/approval-attempt-<n>.json` and
never overwrite. A retry that clobbers the first attempt leaves a pack that
looks like a clean single-approval run, hiding the fact that a box was live and
unapproved in between.
```bash
WORKFLOW=.github/workflows/cmux-tui-testbox-warmup.yml
JOB=cmux-tui-rust
OUT_ROOT="${OUT_ROOT:?set by the exact-source preflight above}"
OUT="$OUT_ROOT"
TBX=""
warmup_testbox_id=""
cleanup_token=""
before_list_status=125
mkdir -p "$OUT/raw"
cleanup() {
local result=$?
local cleanup_status=0
local after_list_status=125
trap - EXIT
if [[ -n "$TBX" && -n "$cleanup_token" && -n "${CONFIRM_TESTBOX_STOP_SHA:-}" ]]; then
set +e
scripts/blacksmith-testbox-cleanup.sh "$TBX" "$OUT" "$cleanup_token" "STOP:${CONFIRM_TESTBOX_STOP_SHA}"
cleanup_status=$?
set -e
else
# Without the CLI receipt there is no proof that a newly listed box belongs
# to this invocation. Report inventory, but never stop another operator's box.
set +e
./scripts/blacksmith-bounded-command.sh 60 \
blacksmith testbox list --all >"$OUT/list-at-exit.log" 2>&1
after_list_status=$?
set -e
if (( after_list_status != 0 )); then
cleanup_status="$after_list_status"
echo "could not capture post-failure Testbox inventory" >&2
else
cleanup_status=1
# Say which of the two conditions actually held. Reporting "no receipt"
# when the receipt exists sends the operator hunting for the wrong thing,
# and the usual cause is simply that no stop was ever authorized.
if [[ -z "${TBX:-}" ]]; then
echo "no Testbox was created; nothing to stop" >&2
elif grep -q "$TBX" "$OUT/list-at-exit.log" 2>/dev/null; then
# Only claim the box is alive if the inventory just said so. Saying it
# after a completed stop ceremony reads as a failure on a clean run.
echo "Testbox ${TBX} is still running; no stop was authorized, so stop it yourself with the PREVIEW then STOP ceremony" >&2
else
echo "no owned Testbox receipt; refusing to stop a box this run cannot prove it owns" >&2
fi
fi
fi
if (( result == 0 && cleanup_status != 0 )) && [[ -n "${CONFIRM_TESTBOX_STOP_SHA:-}" ]]; then
result="$cleanup_status"
fi
exit "$result"
}
trap cleanup EXIT
blacksmith auth whoami 2>&1 | tee "$OUT/whoami.txt" # whoami writes to stderr
blacksmith --version >"$OUT/blacksmith-version.txt"
cat "$OUT/blacksmith-version.txt"
blacksmith runners catalog >"$OUT/runner-catalog.json"
set +e
./scripts/blacksmith-bounded-command.sh 60 \
blacksmith testbox list --all >"$OUT/list-before-warmup.log" 2>&1
before_list_status=$?
set -e
cat "$OUT/list-before-warmup.log"
if (( before_list_status != 0 )); then
echo "refusing to warm a Testbox without a baseline inventory" >&2
exit "$before_list_status"
fi
# Snapshot every gate already waiting in this lane BEFORE dispatching. Set
# difference against this is exact; a time window is not, because a second
# operator dispatching seconds after you lands inside any window you pick.
lane_runs_url="repos/manaflow-ai/cmux/actions/workflows/cmux-tui-testbox-warmup.yml/runs?event=workflow_dispatch&status=waiting"
waiting_before="$(mktemp)"
gh api "$lane_runs_url" --jq '.workflow_runs[].id' | sort >"$waiting_before"
set +e
./scripts/blacksmith-bounded-command.sh 1200 \
blacksmith testbox warmup "$WORKFLOW" \
--ref main \
--job "$JOB" \
--idle-timeout 30 \
>"$OUT/warmup.log" 2>&1
warmup_status=$?
set -e
cat "$OUT/warmup.log"
if (( warmup_status != 0 )); then
# Do not parse IDs from a failed CLI transcript. It may contain a stale ID
# from an error message, and cleanup is intentionally receipt-bound.
exit "$warmup_status"
fi
set +e
warmup_testbox_id="$(python3 - "$OUT/warmup.log" <<'PY'
import re
import sys
text = open(sys.argv[1], encoding="utf-8").read()
ids = re.findall(r"\btbx_[A-Za-z0-9_-]+\b", text)
if not ids:
raise SystemExit("warmup output did not contain a Testbox ID")
print(ids[-1])
PY
)"
parse_status=$?
set -e
if (( parse_status != 0 )); then
exit "$parse_status"
fi
umask 077
set +e
cleanup_token="$(python3 - "$OUT/testbox-receipt.json" "$warmup_testbox_id" "$WORKFLOW" "$JOB" main "$SOURCE_REF" "$SOURCE_SHA" "$SOURCE_TREE_SHA" "$GHOSTTY_SHA" <<'PY'
import datetime as dt
import json
import pathlib
import secrets
import sys
path, testbox_id, workflow, job, warmup_ref, source_ref, source_sha, source_tree, ghostty_sha = sys.argv[1:]
token = secrets.token_hex(16)
path = pathlib.Path(path)
path.write_text(json.dumps({
"schema": 2,
"testbox_id": testbox_id,
"workflow": workflow,
"job": job,
# What the inventory shows, always main in the broker lane.
"warmup_ref": warmup_ref,
# The branch being benchmarked, which never appears in the inventory.
"source_ref": source_ref,
"source_sha": source_sha,
"source_tree_sha": source_tree,
"ghostty_gitlink_sha": ghostty_sha,
"confirmation_token": token,
"created_at": dt.datetime.now(dt.timezone.utc).isoformat(),
}, indent=2, sort_keys=True) + "\n", encoding="utf-8")
path.chmod(0o600)
print(token)
PY
)"
receipt_status=$?
set -e
if (( receipt_status != 0 )); then
echo "could not create the warmup ownership receipt" >&2
exit "$receipt_status"
fi
TBX="$warmup_testbox_id"
printf 'Testbox ID: %s\n' "$TBX" | tee "$OUT/testbox-id.txt"
# Approve the deployment gate BEFORE waiting. The run parks before its first
# step, so `status --wait` below would otherwise burn its full 15 minutes and
# leave this warmed box running.
# Your run is the one that appeared since the snapshot. GitHub does not surface
# it instantly, so poll; zero new runs means "not yet".
waiting_now="$(mktemp)"
approval_run=""
for attempt in $(seq 1 30); do
gh api "$lane_runs_url" --jq '.workflow_runs[].id' | sort >"$waiting_now"
new_runs="$(comm -13 "$waiting_before" "$waiting_now")"
new_count="$(printf '%s' "$new_runs" | grep -c . || true)"
if (( new_count == 1 )); then
approval_run="$new_runs"
break
fi
if (( new_count > 1 )); then
# Two operators dispatched between polls. Nothing in the REST API binds a
# run to a Testbox ID, so do not guess and do not correlate by timestamp:
# Blacksmith rewrites a box's CREATED value as it hydrates. Stop your box,
# re-snapshot, and dispatch again; the next set difference is unambiguous.
echo "$new_count runs appeared at once; stop your box ($TBX), re-snapshot, and re-dispatch" >&2
exit 1
fi
sleep 5
done
if [[ -z "$approval_run" ]]; then
echo "no run appeared within 150s; Testbox $TBX is running and you own it" >&2
exit 1
fi
approval_env="$(gh api "repos/manaflow-ai/cmux/actions/runs/$approval_run/pending_deployments" --jq '.[0].environment.id')"
gh api -X POST "repos/manaflow-ai/cmux/actions/runs/$approval_run/pending_deployments" \
--input - >"$OUT/approval-attempt-1.json" 2>&1 \
<<< "{\"environment_ids\":[$approval_env],\"state\":\"approved\",\"comment\":\"benchmark warmup\"}"
printf 'approved run: %s\n' "$approval_run" | tee "$OUT/approval-run-id.txt"
set +e
blacksmith testbox status --id "$TBX" --wait --wait-timeout 15m \
>"$OUT/status-ready.log" 2>&1
status_ready=$?
set -e
cat "$OUT/status-ready.log"
if (( status_ready != 0 )); then
exit "$status_ready"
fi
```
The workflow refuses any dispatch ref except `refs/heads/main`, and it repeats
that check after `begin-testbox` so a ref that changed during reviewer approval
fails closed. `testbox_id` is the only input, so no caller can steer the job at
a different revision, and `main` alone decides what code runs beside the auth
token.
The workflow concurrency group serializes setup requests by Testbox ID, even
when source SHAs differ. The remote `flock` begins after Blacksmith's rsync, so
it protects stage/build/artifact writes only. Blacksmith exposes no pre-rsync
lease; one Testbox ID must have one owning worktree/operator, and independent
clients must not issue concurrent `run` or download commands. This is an
explicit trusted-lane limitation.
Do not issue a separate interpolated identity command. The setup job's
`setup-identity.json` artifact and each stage helper's pre-build JSON record are
the identity transcripts. They describe different commits on purpose: the setup
artifact names the hydrated `main`, and the stage record names the benchmarked
revision you synchronized. The helper verifies the Testbox VM marker, claimed
Testbox ID, the hydration marker's own internal consistency and runner class,
and then the synchronized source commit/tree, Ghostty gitlink/checkout, and
clean status before it invokes Cargo, repeating the source checks after the
build. Each stage JSON carries a `hydration` block with the warmed ref and
commit plus `matches_benchmarked_source`, which is normally `false`. Keep the
setup artifact URL or download it into `$OUT`. The workflow uploads it under the
name `cmux-tui-testbox-setup-<run-id>`:
```bash
gh run download "$approval_run" --repo manaflow-ai/cmux \
--name "cmux-tui-testbox-setup-$approval_run" --dir "$OUT/setup-artifact"
test -s "$OUT/setup-artifact/setup-identity.json"
```
A successful setup copies the
same JSON to `/tmp/.testbox/cmux-tui-rust-setup-identity.json`; the stage helper
rejects a missing or malformed marker, so failed hydration cannot be
benchmarked. The active Rust, Cargo, and Zig versions must still equal the
hydrated ones, so a branch that repins its toolchain stops the run instead of
reporting a cold-cache timing.
## Pin the box, then take the three timings
Pin the box to the benchmarked commit once, after readiness and before the
first stage. `blacksmith testbox run` synchronizes file contents rather than
history, and skips even that once fingerprints match, so the box otherwise keeps
the `main` checkout the warmup job made:
```bash
./scripts/blacksmith-bounded-command.sh 300 \
blacksmith testbox run --id "$TBX" \
"set -euo pipefail; git fetch --no-tags origin $SOURCE_SHA; git reset --hard $SOURCE_SHA; git submodule update --init --depth 1 ghostty; git rev-parse HEAD" \
>"$OUT/pin-source.log" 2>&1
cat "$OUT/pin-source.log"
```
The commit must already be pushed. A local-only commit fails on the box with
`upload-pack: not our ref`, and the stage helper refuses rather than
benchmarking `main` under a candidate's name.
The warmup job only ever checks out `main`, so this pin is what makes the box an
exact checkout of the revision you are benchmarking. Do it before the stage loop
below. Running the loop first measures `main`, not your branch.
## Three remote build timings
The helper creates one structured JSON record, one raw Cargo log, and one raw
`/usr/bin/time -p` file per stage. Each remote Cargo build is bounded to 20
minutes with a 30-second kill grace period, and the outer CLI invocation is
bounded to 25 minutes so rsync, SSH, or control-plane hangs cannot bypass the
benchmark's cleanup path. It verifies source and submodule identity before the
stage, exports and records the exact Zig binary used by Cargo, holds a remote
`flock` through all writes, restores the controlled changed file from an
integrity-checked backup, and verifies clean identity again.
```bash
# Run this orchestration block in Bash, not an interactive zsh session.
run_stage() {
local stage="$1"
local run_status download_status=0
set +e
printf -v remote_command \
'CMUX_TESTBOX_REMOTE=1 CMUX_TESTBOX_ID=%q %q %q %q %q' \
"$TBX" ./scripts/blacksmith-cmux-tui-testbox-stage.sh \
"$stage" "$SOURCE_SHA" "$GHOSTTY_SHA"
# The second clock. The stage record's wall_seconds is measured on the box
# around cargo; this one includes sync, transport, and queueing, and the gap
# between them is the Testbox overhead.
cli_start="$(python3 -c 'import time; print(time.time())')"
./scripts/blacksmith-bounded-command.sh 1500 blacksmith testbox run --id "$TBX" --debug \
"$remote_command" >"$OUT/$stage.run.log" 2>&1
run_status=$?
set -e
python3 -c "import sys; print(round(float(sys.argv[2]) - float(sys.argv[1]), 3))" \
"$cli_start" "$(python3 -c 'import time; print(time.time())')" >"$OUT/$stage.cli-wall.txt"
cat "$OUT/$stage.run.log"
# rsync --delete can remove remote output before the next run. Download each
# stage immediately, before starting another stage.
: >"$OUT/$stage.download.log"
for suffix in json time log; do
if ! ./scripts/blacksmith-bounded-command.sh 120 \
blacksmith testbox download --id "$TBX" \
"testbox-benchmark/$stage.$suffix" "$OUT/raw/$stage.$suffix" \
>>"$OUT/$stage.download.log" 2>&1; then
download_status=1
fi
done
cat "$OUT/$stage.download.log"
if (( run_status != 0 )); then
return "$run_status"
fi
return "$download_status"
}
benchmark_status=0
for stage in first-clean incremental-noop changed-file; do
if ! assert_source_unchanged; then
benchmark_status=1
break
fi
if ! run_stage "$stage"; then
benchmark_status=1
break
fi
done
if (( benchmark_status != 0 )); then
exit "$benchmark_status"
fi
```
`first-clean` is target-clean but dependency-warm. `incremental-noop` repeats
the exact build on the same VM. `changed-file` appends a comment to
`cmux-tui/crates/cmux-tui/src/main.rs`, builds, and restores the original bytes.
The local worktree is never mutated by the helper.
Verify the downloaded records before accepting timings:
```bash
python3 - "$OUT" "$SOURCE_SHA" "$GHOSTTY_SHA" "$TBX" <<'PY'
import json
import pathlib
import subprocess
import sys
out = pathlib.Path(sys.argv[1])
expected_source, expected_ghostty, testbox_id = sys.argv[2:]
expected_tree = subprocess.check_output(
["git", "rev-parse", f"{expected_source}^{{tree}}"], text=True
).strip()
required = {"first-clean", "incremental-noop", "changed-file"}
records = [json.loads(path.read_text(encoding="utf-8")) for path in sorted((out / "raw").glob("*.json"))]
if {record.get("stage") for record in records} != required:
raise SystemExit("expected exactly three stage records")
for record in records:
stage = record.get("stage")
runner = record.get("runner", {})
if runner.get("arch") != "x86_64" or runner.get("cpu_count") != 32:
raise SystemExit(f"{stage}: wrong runner identity {runner}")
if record.get("testbox", {}).get("id") != testbox_id:
raise SystemExit(f"{stage}: wrong Testbox ID")
source_record = record.get("source", {})
if source_record.get("expected_commit_sha") != expected_source or source_record.get("expected_tree_sha") != expected_tree:
raise SystemExit(f"{stage}: wrong expected source identity")
for side in ("before", "after"):
source = source_record.get(side, {})
if source.get("commit_sha") != expected_source or source.get("tree_sha") != expected_tree or source.get("dirty_files"):
raise SystemExit(f"{stage}: source mismatch or dirty {side} checkout")
ghostty = source.get("ghostty", {})
if ghostty.get("gitlink_sha") != expected_ghostty or ghostty.get("head_sha") != expected_ghostty or ghostty.get("dirty_files"):
raise SystemExit(f"{stage}: Ghostty mismatch or dirty {side} checkout")
if not record.get("ok"):
raise SystemExit(f"{stage}: build failed")
with (out / "timings.json").open("w", encoding="utf-8") as handle:
json.dump({"schema": 2, "stage_record_schema": 3, "source_sha": expected_source, "ghostty_gitlink_sha": expected_ghostty, "testbox_id": testbox_id, "stages": records}, handle, indent=2, sort_keys=True)
handle.write("\n")
PY
```
## Cleanup and evidence
Stopping the box is only half of cleanup. The warmup run's keepalive step keeps
holding a 32 vCPU runner afterwards, with a 120 minute job timeout as the only
backstop, so cancel the run too and poll it to terminal state:
```bash
gh run cancel "$approval_run" --repo manaflow-ai/cmux >"$OUT/run-cancel.log" 2>&1 || true
for attempt in $(seq 1 40); do
run_state="$(gh api "repos/manaflow-ai/cmux/actions/runs/$approval_run" --jq '"\(.status) \(.conclusion)"')"
printf '%s %s\n' "$(date -u +%FT%TZ)" "$run_state" >>"$OUT/run-cancel-poll.log"
[[ "$run_state" == completed* ]] && break
sleep 15
done
printf 'final run state: %s\n' "$run_state" | tee "$OUT/run-final-state.txt"
```
Cancelling takes about five minutes to land, so `in_progress` right after the
request is expected, not a failed cancel.
Download all raw files and `timings.json` before cleanup. Then, after an
operator explicitly decides this exact box may be destroyed, call the fail-safe
helper. It previews the exact receipt context, preserves an already-completed
409, polls cancellation to a bounded deadline, fails on other stop/status/list
errors, and verifies this exact Testbox ID is no longer active:
```bash
cleanup_token="${cleanup_token:-}"
[[ "$cleanup_token" =~ ^[0-9a-f]{32}$ ]] || {
echo "use the confirmation token emitted by the warmup receipt" >&2
exit 64
}
# PREVIEW exits 75 on success, meaning "preview written, nothing destroyed
# yet". Run it outside `set -e`, which this plan otherwise enables, or it aborts
# the orchestration immediately before cleanup.
set +e
scripts/blacksmith-testbox-cleanup.sh "$TBX" "$OUT" "$cleanup_token" PREVIEW
preview_status=$?
set -e
(( preview_status == 75 )) || { echo "cleanup preview failed with $preview_status" >&2; exit "$preview_status"; }
# Review cleanup-preview.json, then rerun with STOP:<sha256(cleanup-preview.json)>.
```
A shell `EXIT` trap may call that helper only when an independent operator has
exported `CONFIRM_TESTBOX_STOP_SHA` with the SHA-256 of a separately reviewed
`cleanup-preview.json`; otherwise it preserves the benchmark status, records
inventory, and leaves the box for explicit manual cleanup.
Warmup writes `testbox-receipt.json` and an ownership token bound to the exact
returned ID; cleanup refuses a mismatched ID or token. If warmup fails before
returning an ID, retain before/after inventories but do not automatically stop a
box, because an inventory diff cannot prove ownership across concurrent
operators. Reconcile that orphan manually through the
Blacksmith control plane. Keep both inventories and their command statuses,
plus warmup/status/identity transcripts, every stage run and download
transcript, raw JSON/time/log files, runner catalog, setup identity artifact,
cleanup logs, the receipt, and the final source manifest in the new, unique
`.cmux-scratch/` directory. Never reuse a prior SHA-only directory or overwrite
historical records. Never store credentials, private keys, or
`/tmp/.testbox/auth_token`.
Record these fields alongside `timings.json`:
1. Exact source branch, full source SHA/tree SHA, Ghostty gitlink SHA, and the
clean-status result before each stage.
2. Requested runner label and catalog output. The setup job rejects any
actual architecture or CPU count other than x64 and 32.
3. Blacksmith CLI version, Testbox ID, and the setup workflow run and job IDs.
There is no separate identity run: this plan forbids issuing one, and the
identity transcripts are the setup artifact plus each stage's own record.
4. Whether the comparison was target-clean, registry/git-cache warm,
Zig-cache warm, or a genuinely cold VM. Warmup deliberately hydrates
dependencies, so `first-clean` is target-cold and dependency-warm.
5. Cleanup stop/status/list output and whether the specific ID was absent from
the active inventory.
Never write into an evidence directory you did not create in this run. The
`-e "$OUT_ROOT"` check above is that guard: if the path exists, stop and choose a
new run suffix rather than merging two runs' records into one pack.
Prior hosted cmux-tui correctness runs without Cargo durations are provenance,
not performance comparisons. Prior Blacksmith macOS Swift/Xcode artifacts use
a different OS, architecture, runner SKU, cache state, and workload, so they
are context rather than a Rust baseline.
skills/blacksmith-testbox/references/operations.md
-225
View File
@@ -1,225 +0,0 @@
# Testbox operations reference
Long-form detail for `skills/blacksmith-testbox/SKILL.md`: the stage
orchestration loop, the receipt-bound cleanup ceremony, and how to read the two
clocks a run produces. The full evidence-producing plan lives in
`benchmark.md`.
## Remote benchmark stages
The detailed, receipt-producing orchestration in `benchmark.md` is the required
entry point for a complete benchmark. It creates the unique `OUT_ROOT`, receipt,
cleanup token, setup artifact capture, and cleanup preview state. Do not copy
only this stage loop into an ad hoc shell without those prerequisites.
Before each stage, recompute `SOURCE_SHA` and `GHOSTTY_SHA` and repeat the
clean pushed-branch preflight.
Pass the expected values as validated arguments;
the helper does not trust the remote checkout or a caller-supplied expected SHA
without comparing it to Git metadata:
```bash
# Run this orchestration block in Bash, not an interactive zsh session.
run_stage() {
local stage="$1"
local run_status download_status=0
set +e
printf -v remote_command \
'CMUX_TESTBOX_REMOTE=1 CMUX_TESTBOX_ID=%q %q %q %q %q' \
"$TBX" ./scripts/blacksmith-cmux-tui-testbox-stage.sh \
"$stage" "$SOURCE_SHA" "$GHOSTTY_SHA"
./scripts/blacksmith-bounded-command.sh 1500 \
blacksmith testbox run --id "$TBX" --debug \
"$remote_command" >"$OUT/$stage.run.log" 2>&1
run_status=$?
set -e
cat "$OUT/$stage.run.log"
# Download immediately. Blacksmith's next rsync may delete or replace remote
# files, so a one-time download after all stages is insufficient.
: >"$OUT/$stage.download.log"
for suffix in json time log; do
if ! ./scripts/blacksmith-bounded-command.sh 120 \
blacksmith testbox download --id "$TBX" \
"testbox-benchmark/$stage.$suffix" "$OUT/raw/$stage.$suffix" \
>>"$OUT/$stage.download.log" 2>&1; then
download_status=1
fi
done
cat "$OUT/$stage.download.log"
if (( run_status != 0 )); then
return "$run_status"
fi
return "$download_status"
}
```
The helper supports exactly `first-clean`, `incremental-noop`, and
`changed-file`. Each remote Cargo build is bounded to 20 minutes with a
30-second kill grace period. It records a schema-3 JSON object for each stage
containing:
* expected and observed source commit/tree identity before and after the build;
* expected and observed Ghostty gitlink and initialized submodule HEAD;
* clean/dirty file lists and source restoration status;
* Testbox ID and adopted workflow run ID;
* runner label, hostname, architecture, CPU count, and `uname`; the setup
workflow fails closed unless the actual runner is x64 with 32 CPUs;
* active Rust toolchain, `rustc`, Cargo, Zig, lockfile/toolchain hashes, and
Ghostty package-manifest hash; and
* Cargo exit status, `/usr/bin/time -p` values, and CLI transcript timing.
`first-clean` removes only the remote `cmux-tui/target` directory before a
`cargo build -p cmux-tui --locked`. `incremental-noop` repeats that command
without changing source. `changed-file` appends a comment to
`cmux-tui/crates/cmux-tui/src/main.rs`, builds, and restores the original bytes
before emitting its final record. A dirty or mismatched source before any
stage, after restoration, or in the Ghostty submodule aborts the stage.
After all successful downloads, aggregate and verify the records:
```bash
python3 - "$OUT" "$SOURCE_SHA" "$GHOSTTY_SHA" "$TBX" <<'PY'
import json
import pathlib
import subprocess
import sys
out = pathlib.Path(sys.argv[1])
expected_source, expected_ghostty, testbox_id = sys.argv[2:]
expected_tree = subprocess.check_output(
["git", "rev-parse", f"{expected_source}^{{tree}}"], text=True
).strip()
required = {"first-clean", "incremental-noop", "changed-file"}
records = []
for path in sorted((out / "raw").glob("*.json")):
record = json.loads(path.read_text(encoding="utf-8"))
records.append(record)
if {record.get("stage") for record in records} != required:
raise SystemExit("timing evidence is missing one or more benchmark stages")
for record in records:
if record.get("testbox", {}).get("id") != testbox_id:
raise SystemExit(f"{record.get('stage')} has the wrong Testbox ID")
source = record.get("source", {})
if source.get("expected_commit_sha") != expected_source or source.get("expected_tree_sha") != expected_tree:
raise SystemExit(f"{record.get('stage')} has the wrong expected source identity")
for side in ("before", "after"):
snapshot = source.get(side, {})
if snapshot.get("commit_sha") != expected_source or snapshot.get("tree_sha") != expected_tree:
raise SystemExit(f"{record.get('stage')} has the wrong {side} source SHA")
if snapshot.get("dirty_files"):
raise SystemExit(f"{record.get('stage')} has dirty top-level source")
ghostty = snapshot.get("ghostty", {})
if ghostty.get("gitlink_sha") != expected_ghostty or ghostty.get("head_sha") != expected_ghostty:
raise SystemExit(f"{record.get('stage')} has mismatched Ghostty identity")
if ghostty.get("dirty_files"):
raise SystemExit(f"{record.get('stage')} has dirty Ghostty source")
if not record.get("ok"):
raise SystemExit(f"{record.get('stage')} did not complete successfully")
with (out / "timings.json").open("w", encoding="utf-8") as handle:
json.dump({"schema": 2, "stage_record_schema": 3, "source_sha": expected_source, "ghostty_gitlink_sha": expected_ghostty, "testbox_id": testbox_id, "stages": records}, handle, indent=2, sort_keys=True)
handle.write("\n")
PY
```
Keep `raw/*.json`, `raw/*.time`, `raw/*.log`, every `*.run.log` and download
log, the setup artifact, and the source manifest in a new, unique
`.cmux-scratch/` evidence directory. Never reuse a prior SHA-only directory;
refuse to overwrite historical records. Do not add credentials or private keys.
## Fail-safe cleanup
Always download before cleanup. Use the checked-in cleanup helper rather than
ignoring errors with `|| true`. After an independent operator decides the exact
box may be destroyed, pass the ownership token generated with the warmup receipt
and the literal `STOP`; never print the token:
```bash
CLEANUP_TOKEN="${CLEANUP_TOKEN:-}"
[[ "$CLEANUP_TOKEN" =~ ^[0-9a-f]{32}$ ]] || {
echo "use the ownership token emitted by the warmup receipt" >&2
exit 64
}
scripts/blacksmith-testbox-cleanup.sh "$TBX" "$OUT" "$CLEANUP_TOKEN" PREVIEW # exits 75 on success: preview written, nothing destroyed
# Review cleanup-preview.json, then rerun with STOP:<sha256(cleanup-preview.json)>.
```
It records a pre-stop status preview, stop result, post-stop status, and
`list --all` output. The preview must match the receipt's workflow, job, and
branch before any stop is attempted. Cleanup is destructive and requires a fresh `STOP:<sha256(cleanup-preview.json)>`
confirmation after reviewing the current receipt-bound preview.
It verifies that the specific Testbox ID is terminal or absent from the active
inventory, accepts the known terminal states `completed`, `stopped`, `cancelled`,
`failed`, `terminated`, and `hydration_failed`, plus a 409 saying the box is
already stopped or completed, and polls for up to two minutes while cancellation
propagates. Other stop, status, or list failures remain failures.
Put it in an `EXIT` trap only after an independent operator exports
`CONFIRM_TESTBOX_STOP_SHA` containing the SHA-256 of a separately reviewed
`cleanup-preview.json`; otherwise preserve the benchmark's original exit status
and leave the box for manual cleanup. The detailed benchmark writes a
receipt and ownership token for the exact ID returned by warmup; cleanup refuses an ID
or token that is not bound to that receipt. If warmup fails before returning an
ID, retain before/after inventory but do not automatically stop a box, because
an inventory diff cannot prove ownership across concurrent operators. Reconcile
that orphan manually through the Blacksmith control plane.
## Partial stage sets
The verification and aggregation blocks require all three stages and raise
`SystemExit` on any subset, because a `timings.json` that silently omits a stage
reads as a complete result. Running one or two stages is fine, and common: read
the per-stage `testbox-benchmark/<stage>.json` records directly and do not
produce a `timings.json` at all.
## Timing interpretation
The benchmark reports two clocks, and the stage record contains only one of
them. `wall_seconds` in each stage JSON is measured **on the box**, around the
cargo command, which is why it sits a few milliseconds above the
`/usr/bin/time -p` `real` value rather than well above it. It is not the CLI
clock.
The second clock is local: the wall time of the `blacksmith testbox run`
invocation itself, which also covers sync, transport, and queueing. The plan in
`benchmark.md` writes it to `$OUT/<stage>.cli-wall.txt`. Compare remote `real`
or `time_real_seconds` for build performance, and use the difference between the
local and remote clocks for Testbox overhead. A pack without `cli-wall.txt`
cannot measure overhead at all.
Compare overhead only across `first-clean` and `incremental-noop`. The
`changed-file` gap is unstable across runs, measured at 8.1 s in one and 1.48 s
in another with no change to the lane, because that stage's backup, edit,
restore, and re-verify work runs inside the CLI call but outside `wall_seconds`.
Its local-minus-remote figure is real but is not sync and transport, so do not
read it as Testbox overhead and do not expect a repeatable magnitude.
`first-clean` is target-clean but dependency-warm: warmup runs `cargo fetch` and
`zig build --fetch`, and the workflow may restore registry, git, and Zig caches.
`incremental-noop` measures a second build on the same VM. `changed-file` is a
controlled source change on that same VM. These are deliberately different
from a cold-VM benchmark.
Treat any evidence directory you did not create this run as read-only. Never
select an existing one as a writable `OUT`, and never rewrite its raw records or
cleanup result.
After a stop, `blacksmith testbox status --id <id>` still prints a row reading
`completed` while `list --all` reports no active testboxes. Both are correct:
`status` answers about one ID including terminal ones, and `list` shows only
live boxes. Cite both, and do not read the difference as a failed stop.
The two identity transcripts spell the architecture differently for the same
host: `setup-identity.json` records GitHub's `RUNNER_ARCH` (`X64`), while stage
records use `uname` (`x86_64`). The verification block asserts the `uname`
spelling. This is expected, not a mismatch.
**Never pair a box with a workflow run by timestamp.** Blacksmith's `CREATED`
column is not a creation time. It tracks the last state
transition, so one box reports a different value while hydrating, at ready, and
after stop. For elapsed hydration read the `status --wait` transcript, which
prints the wait duration next to `Testbox ready!`. One measured box reported `05:54:43`,
then `05:55:03`, then `05:58:48` as it hydrated, so a box whose value looks
adjacent to a run's creation time is coincidence past the `queued` state. The
authoritative binding appears only once the box is ready, as the RUN URL column
of `blacksmith testbox list --all`. Before that, identify your run by set
difference against a pre-dispatch snapshot.
skills/blacksmith-testbox/references/trust-boundary.md
-25
View File
@@ -1,25 +0,0 @@
# Why this lane hydrates main only
Full reasoning behind the `--ref main` rule in `skills/blacksmith-testbox/SKILL.md`.
Read this before changing the warmup workflow, the environment configuration, or
anything that runs before `begin-testbox`.
`useblacksmith/begin-testbox` writes `/tmp/.testbox/auth_token` into the CI job,
and `permissions: contents: read` does not stop a later step from reading it.
`blacksmith testbox warmup` resolves the workflow definition and the hydrated
source from the same `--ref`, so warming a candidate branch would run that
branch's copy of the workflow beside the token, and the branch could delete its
own guards.
The lane hydrates `main` only. The first step refuses any other ref, no
repository code runs before the token, and your revision arrives afterwards
through `blacksmith testbox run` as an authenticated org member who could
already reach the box. `tests/test_ci_testbox_broker_guard.py` enforces that
shape on every pull request through
`.github/workflows/testbox-broker-guard.yml`.
A repository administrator owns the `blacksmith-testbox-trusted` environment:
required reviewers, no secrets, admin bypass disabled, and a deployment branch
rule of exactly `main`. If that drifts, disable the lane and stop. Never edit
the workflow to work around a missing control.
tests/test_testbox_doc_blocks.sh
-73
View File
@@ -1,73 +0,0 @@
#!/usr/bin/env bash
# The Testbox plan is executable documentation: agents copy its shell blocks and
# run them under the `set -euo pipefail` the plan itself mandates. Two separate
# defects have shipped because a block was syntactically fine but died at
# runtime, each time stranding a live 32 vCPU box:
#
# * `grep -c .` exits 1 when the count is zero, so a poll loop that expects
# "not visible yet" as its normal first state killed the whole script.
#
# Syntax checking cannot see that. This test parses every documented block and
# executes the fragile constructs for real.
set -euo pipefail
root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
skill_dir="$root/skills/blacksmith-testbox"
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT
files=("$skill_dir/SKILL.md" "$skill_dir/benchmark.md" "$skill_dir/references/operations.md")
for file in "${files[@]}"; do
test -f "$file" || { echo "FAIL: missing $file" >&2; exit 1; }
done
# 1. Every fenced bash block must parse under the shell the plan mandates.
blocks=0
while IFS= read -r block; do
blocks=$((blocks + 1))
printf 'set -euo pipefail\n%s\n' "$(cat "$block")" >"$work/block.sh"
if ! bash -n "$work/block.sh" 2>"$work/err"; then
echo "FAIL: $(basename "$block") does not parse:" >&2
cat "$work/err" >&2
exit 1
fi
done < <(python3 - "$work" "${files[@]}" <<'PYX'
import pathlib
import re
import sys
out = pathlib.Path(sys.argv[1])
index = 0
for name in sys.argv[2:]:
text = pathlib.Path(name).read_text(encoding="utf-8")
for body in re.findall(r"```bash\n(.*?)```", text, re.S):
index += 1
path = out / f"{pathlib.Path(name).stem}-{index}.sh"
path.write_text(body, encoding="utf-8")
print(path)
PYX
)
test "$blocks" -gt 0 || { echo "FAIL: no bash blocks found; the parser broke" >&2; exit 1; }
# 2. Every counting construct must survive its normal zero case under set -e.
# Extract each line that counts with grep -c and run it with an empty input.
found_counts=0
while IFS= read -r line; do
found_counts=$((found_counts + 1))
cat >"$work/count.sh" <<COUNT
set -euo pipefail
WAITING=""
waiting=""
$line
echo "survived"
COUNT
if ! output="$(bash "$work/count.sh" 2>&1)" || [[ "$output" != *survived* ]]; then
echo "FAIL: this counting line dies under set -e when nothing matches, which" >&2
echo " is its normal first state. Guard it with '|| true'." >&2
echo " $line" >&2
exit 1
fi
done < <(grep -rhE '^[[:space:]]*[A-Za-z_]+=.*grep -c' "${files[@]}" | sed 's/^[[:space:]]*//')
test "$found_counts" -gt 0 || { echo "FAIL: no counting lines found; the extractor broke" >&2; exit 1; }
echo "ok: $blocks documented bash blocks parse, $found_counts counting lines survive an empty result"