ci: warm cmux-tui Testboxes from a main-controlled broker
blacksmith testbox warmup resolves the workflow definition and the hydrated source from one --ref, so a lane that warms a candidate branch runs that branch's copy of the workflow before begin-testbox writes the Testbox auth token into the job. A candidate could therefore delete its own guards. Hydrate main only. The first step refuses any ref except refs/heads/main, and no repository code runs before the token. A candidate revision reaches the box afterwards through blacksmith testbox run, which syncs a maintainer's worktree onto the warm VM and needs Blacksmith org credentials that already grant box access, so it moves no trust boundary. The hydrated commit and the benchmarked commit are now deliberately different. The stage helper checks the setup marker for VM identity, runner class, and toolchain completeness instead of source equality, records the hydrated ref and SHA under a new "hydration" block, and still fails closed when the active Rust, Cargo, or Zig differs from what warmed the caches. The lane no longer needs BLACKSMITH_TESTBOX_REVIEWED_REF or BLACKSMITH_TESTBOX_REVIEWED_SHA; the environment needs a deployment branch rule of exactly main.
This commit is contained in:
Lawrence Chen
@@ -19,4 +19,5 @@ self-hosted-runner:
|
||||
# Linux: Blacksmith primary (LINUX_RUNNER), WarpBuild overflow fallback.
|
||||
- blacksmith-4vcpu-ubuntu-2404
|
||||
- blacksmith-8vcpu-ubuntu-2404
|
||||
- blacksmith-32vcpu-ubuntu-2404
|
||||
- warp-ubuntu-latest-x64-4x
|
||||
|
||||
@@ -0,0 +1,366 @@
|
||||
name: cmux-tui Rust Testbox setup
|
||||
|
||||
# Main-controlled broker. This workflow runs only from refs/heads/main and
|
||||
# hydrates only main, so no candidate branch can edit the guards that run
|
||||
# before begin-testbox exposes its auth token, and no candidate build script
|
||||
# executes inside this token-bearing job. A candidate revision reaches the
|
||||
# Testbox later, through `blacksmith testbox run`, which synchronizes a
|
||||
# maintainer's local worktree onto the already-warm VM.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
testbox_id:
|
||||
description: "Testbox session ID supplied by blacksmith testbox warmup"
|
||||
required: true
|
||||
type: string
|
||||
|
||||
permissions: {}
|
||||
|
||||
concurrency:
|
||||
# A Testbox is a mutable shared workspace. Serialize every request for the
|
||||
# same ID so two dispatches cannot corrupt one cache.
|
||||
group: cmux-tui-testbox-${{ inputs.testbox_id }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
cmux-tui-rust:
|
||||
name: cmux-tui Rust setup
|
||||
runs-on: blacksmith-32vcpu-ubuntu-2404
|
||||
environment:
|
||||
# Configure this environment with required reviewers, no secrets, and a
|
||||
# deployment branch rule of exactly `main`. Approval is evaluated before
|
||||
# the first step, so it precedes begin-testbox.
|
||||
name: blacksmith-testbox-trusted
|
||||
permissions:
|
||||
contents: read
|
||||
# Hydration plus three sequential 20-minute bounded remote builds happen
|
||||
# after setup. Keep the GitHub job alive long enough for all stages and
|
||||
# cleanup; the Testbox itself still has its separate idle timeout.
|
||||
timeout-minutes: 120
|
||||
steps:
|
||||
# Every check here is main-controlled. `blacksmith testbox warmup`
|
||||
# resolves both this file and the hydrated source from the same --ref, so
|
||||
# refusing any ref other than refs/heads/main is what keeps a candidate
|
||||
# branch outside the trust boundary.
|
||||
- name: Validate broker ref
|
||||
env:
|
||||
DISPATCH_REF: ${{ github.ref }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
TESTBOX_ID: ${{ inputs.testbox_id }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
[[ "$REPOSITORY" == "manaflow-ai/cmux" ]] || {
|
||||
echo "::error::this Testbox lane is only valid for manaflow-ai/cmux" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "$EVENT_NAME" == "workflow_dispatch" ]] || {
|
||||
echo "::error::Testbox setup must be dispatched manually, never from a PR event" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "$DISPATCH_REF" == "refs/heads/main" ]] || {
|
||||
echo "::error::this broker lane runs only from refs/heads/main, got $DISPATCH_REF; warm up with --ref main and sync the candidate through blacksmith testbox run" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "$TESTBOX_ID" =~ ^tbx_[A-Za-z0-9_-]+$ ]] || {
|
||||
echo "::error::malformed Testbox ID" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Begin Testbox
|
||||
uses: useblacksmith/begin-testbox@233448af4bfdc6fca509a7f0974411ac6d8a8043 # v2
|
||||
with:
|
||||
testbox_id: ${{ inputs.testbox_id }}
|
||||
|
||||
# main can move while a reviewer approves the deployment. Record which
|
||||
# commit this job actually hydrates and fail closed if the dispatch ref
|
||||
# changed under it.
|
||||
- name: Revalidate broker identity after token exposure
|
||||
env:
|
||||
DISPATCH_REF: ${{ github.ref }}
|
||||
DISPATCH_SHA: ${{ github.sha }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
[[ "$DISPATCH_REF" == "refs/heads/main" ]] || {
|
||||
echo "::error::broker ref changed during setup" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ "$DISPATCH_SHA" =~ ^[0-9a-f]{40}$ ]] || {
|
||||
echo "::error::dispatch SHA is malformed" >&2
|
||||
exit 1
|
||||
}
|
||||
printf 'hydrating main at %s\n' "$DISPATCH_SHA"
|
||||
|
||||
- name: Checkout hydration commit
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
ref: ${{ github.sha }}
|
||||
|
||||
- name: Require exact checkout and clean source
|
||||
env:
|
||||
EXPECTED_SHA: ${{ github.sha }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
actual_sha="$(git rev-parse HEAD)"
|
||||
[[ "$actual_sha" == "$EXPECTED_SHA" ]] || {
|
||||
echo "::error::checked out $actual_sha, expected hydration SHA $EXPECTED_SHA" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ -z "$(git status --porcelain=v1 --untracked-files=normal)" ]] || {
|
||||
echo "::error::source checkout is dirty before hydration" >&2
|
||||
git status --short >&2
|
||||
exit 1
|
||||
}
|
||||
source_tree_sha="$(git rev-parse 'HEAD^{tree}')"
|
||||
ghostty_entry="$(git ls-tree HEAD ghostty)"
|
||||
[[ "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ]] || {
|
||||
echo "::error::HEAD:ghostty is not a gitlink" >&2
|
||||
exit 1
|
||||
}
|
||||
ghostty_gitlink_sha="$(git rev-parse 'HEAD:ghostty')"
|
||||
printf 'source_sha=%s\nsource_tree_sha=%s\nghostty_gitlink_sha=%s\n' \
|
||||
"$actual_sha" "$source_tree_sha" "$ghostty_gitlink_sha"
|
||||
|
||||
- name: Initialize Ghostty source submodule
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
git submodule update --init --depth 1 ghostty
|
||||
[[ "$(git -C ghostty rev-parse --show-toplevel)" == "$GITHUB_WORKSPACE/ghostty" ]] || {
|
||||
echo "::error::ghostty did not initialize as its own submodule checkout" >&2
|
||||
exit 1
|
||||
}
|
||||
ghostty_entry="$(git ls-tree HEAD ghostty)"
|
||||
[[ "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ]] || {
|
||||
echo "::error::HEAD:ghostty is not a gitlink" >&2
|
||||
exit 1
|
||||
}
|
||||
expected_ghostty_sha="$(git rev-parse HEAD:ghostty)"
|
||||
actual_ghostty_sha="$(git -C ghostty rev-parse HEAD)"
|
||||
[[ "$actual_ghostty_sha" == "$expected_ghostty_sha" ]] || {
|
||||
echo "::error::Ghostty checkout $actual_ghostty_sha does not match gitlink $expected_ghostty_sha" >&2
|
||||
exit 1
|
||||
}
|
||||
[[ -z "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]] || {
|
||||
echo "::error::Ghostty submodule is dirty after initialization" >&2
|
||||
git -C ghostty status --short >&2
|
||||
exit 1
|
||||
}
|
||||
test -f ghostty/build.zig.zon
|
||||
|
||||
- name: Install Linux build dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y clang libclang-dev pkg-config
|
||||
|
||||
- name: Cache Zig package downloads
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
with:
|
||||
path: ~/.cache/zig
|
||||
key: cmux-tui-zig-${{ hashFiles('ghostty/build.zig.zon', 'ghostty/build.zig.zon.json') }}
|
||||
restore-keys: |
|
||||
cmux-tui-zig-
|
||||
|
||||
- name: Install repository-pinned Zig
|
||||
shell: bash
|
||||
run: ./scripts/install-zig-ci.sh
|
||||
|
||||
- name: Fetch Ghostty Zig dependencies without compiling
|
||||
working-directory: ghostty
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# `--fetch` hydrates the package cache and exits before a build.
|
||||
"$CMUX_ZIG" build --fetch
|
||||
|
||||
- name: Cache Cargo registry and git dependencies
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
with:
|
||||
path: |
|
||||
~/.cargo/registry
|
||||
~/.cargo/git
|
||||
key: cmux-tui-cargo-${{ runner.os }}-${{ hashFiles('cmux-tui/Cargo.lock', 'cmux-tui/rust-toolchain.toml') }}
|
||||
restore-keys: |
|
||||
cmux-tui-cargo-${{ runner.os }}-
|
||||
|
||||
- name: Set up repository-pinned cmux-tui Rust
|
||||
uses: ./.github/actions/setup-cmux-tui-rust
|
||||
|
||||
- name: Fetch Cargo dependencies without compiling
|
||||
working-directory: cmux-tui
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
cargo fetch --locked
|
||||
|
||||
- name: Record runner, toolchain, and Ghostty identity
|
||||
env:
|
||||
SOURCE_SHA: ${{ github.sha }}
|
||||
SOURCE_REF: ${{ github.ref }}
|
||||
TESTBOX_ID: ${{ inputs.testbox_id }}
|
||||
RUNNER_LABEL: blacksmith-32vcpu-ubuntu-2404
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
source_tree_sha="$(git rev-parse 'HEAD^{tree}')"
|
||||
ghostty_entry="$(git ls-tree HEAD ghostty)"
|
||||
[[ "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ]] || exit 1
|
||||
ghostty_gitlink_sha="$(git rev-parse 'HEAD:ghostty')"
|
||||
ghostty_head_sha="$(git -C ghostty rev-parse HEAD)"
|
||||
[[ "$SOURCE_SHA" == "$(git rev-parse HEAD)" ]] || exit 1
|
||||
[[ "$ghostty_gitlink_sha" == "$ghostty_head_sha" ]] || exit 1
|
||||
[[ -z "$(git status --porcelain=v1 --untracked-files=normal)" ]] || exit 1
|
||||
[[ -z "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]] || exit 1
|
||||
pushd cmux-tui >/dev/null
|
||||
cargo metadata --locked --no-deps --format-version 1 > "$RUNNER_TEMP/cmux-tui-cargo-metadata.json"
|
||||
test -s "$RUNNER_TEMP/cmux-tui-cargo-metadata.json"
|
||||
RUST_TOOLCHAIN="$(rustup show active-toolchain)"
|
||||
RUSTC_VERSION="$(rustc --version)"
|
||||
CARGO_VERSION="$(cargo --version)"
|
||||
popd >/dev/null
|
||||
mkdir -p testbox-benchmark
|
||||
SOURCE_TREE_SHA="$source_tree_sha"
|
||||
GHOSTTY_GITLINK_SHA="$ghostty_gitlink_sha"
|
||||
GHOSTTY_HEAD_SHA="$ghostty_head_sha"
|
||||
ZIG_VERSION="$("$CMUX_ZIG" version)"
|
||||
ZIG_PATH="$CMUX_ZIG"
|
||||
RUNNER_UNAME="$(uname -a)"
|
||||
RUNNER_CPU_COUNT="$(nproc)"
|
||||
[[ "$RUNNER_ARCH" == "X64" && "$RUNNER_CPU_COUNT" == "32" ]] || {
|
||||
echo "::error::expected x64 32-vCPU runner, got arch=$RUNNER_ARCH cpu_count=$RUNNER_CPU_COUNT" >&2
|
||||
exit 1
|
||||
}
|
||||
CARGO_METADATA_SHA256="$(sha256sum "$RUNNER_TEMP/cmux-tui-cargo-metadata.json" | cut -d ' ' -f 1)"
|
||||
RUST_TOOLCHAIN_FILE_SHA256="$(sha256sum cmux-tui/rust-toolchain.toml | cut -d ' ' -f 1)"
|
||||
CARGO_LOCK_SHA256="$(sha256sum cmux-tui/Cargo.lock | cut -d ' ' -f 1)"
|
||||
GHOSTTY_ZON_SHA256="$(sha256sum ghostty/build.zig.zon | cut -d ' ' -f 1)"
|
||||
export SOURCE_TREE_SHA GHOSTTY_GITLINK_SHA GHOSTTY_HEAD_SHA RUST_TOOLCHAIN RUSTC_VERSION CARGO_VERSION ZIG_VERSION ZIG_PATH RUNNER_UNAME RUNNER_CPU_COUNT CARGO_METADATA_SHA256 RUST_TOOLCHAIN_FILE_SHA256 CARGO_LOCK_SHA256 GHOSTTY_ZON_SHA256
|
||||
python3 - <<'PY' > testbox-benchmark/setup-identity.json
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
|
||||
print(json.dumps({
|
||||
"schema": 3,
|
||||
"source": {
|
||||
"ref": os.environ["SOURCE_REF"],
|
||||
"commit_sha": os.environ["SOURCE_SHA"],
|
||||
"tree_sha": os.environ["SOURCE_TREE_SHA"],
|
||||
"ghostty_gitlink_sha": os.environ["GHOSTTY_GITLINK_SHA"],
|
||||
"ghostty_head_sha": os.environ["GHOSTTY_HEAD_SHA"],
|
||||
},
|
||||
"broker": {
|
||||
"workflow_ref": os.environ["GITHUB_REF"],
|
||||
"workflow_sha": os.environ["SOURCE_SHA"],
|
||||
},
|
||||
"testbox": {
|
||||
"id": os.environ["TESTBOX_ID"],
|
||||
"setup_workflow_run_id": os.environ["GITHUB_RUN_ID"],
|
||||
},
|
||||
"runner": {
|
||||
"label": os.environ["RUNNER_LABEL"],
|
||||
"name": os.environ.get("RUNNER_NAME"),
|
||||
"os": os.environ.get("RUNNER_OS"),
|
||||
"arch": os.environ.get("RUNNER_ARCH"),
|
||||
"hostname": platform.node(),
|
||||
"uname": os.environ["RUNNER_UNAME"],
|
||||
"cpu_count": int(os.environ["RUNNER_CPU_COUNT"]),
|
||||
},
|
||||
"toolchain": {
|
||||
"rust_toolchain": os.environ["RUST_TOOLCHAIN"],
|
||||
"rustc": os.environ["RUSTC_VERSION"],
|
||||
"cargo": os.environ["CARGO_VERSION"],
|
||||
"rust_toolchain_file_sha256": os.environ["RUST_TOOLCHAIN_FILE_SHA256"],
|
||||
"cargo_lock_sha256": os.environ["CARGO_LOCK_SHA256"],
|
||||
"cargo_metadata_sha256": os.environ["CARGO_METADATA_SHA256"],
|
||||
"zig_path": os.environ["ZIG_PATH"],
|
||||
"zig": os.environ["ZIG_VERSION"],
|
||||
"ghostty_build_zig_zon_sha256": os.environ["GHOSTTY_ZON_SHA256"],
|
||||
},
|
||||
}, sort_keys=True, indent=2))
|
||||
PY
|
||||
test -s testbox-benchmark/setup-identity.json
|
||||
cat testbox-benchmark/setup-identity.json
|
||||
|
||||
- name: Require clean hydrated source
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
[[ -z "$(git status --porcelain=v1 --untracked-files=normal)" ]] || {
|
||||
echo "::error::source became dirty during hydration" >&2
|
||||
git status --short >&2
|
||||
exit 1
|
||||
}
|
||||
[[ -z "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]] || {
|
||||
echo "::error::Ghostty became dirty during hydration" >&2
|
||||
git -C ghostty status --short >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
- name: Upload setup identity JSON
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: cmux-tui-testbox-setup-${{ github.run_id }}
|
||||
path: testbox-benchmark/setup-identity.json
|
||||
if-no-files-found: warn
|
||||
retention-days: 14
|
||||
|
||||
# Missing `/tmp/.testbox` means begin-testbox could not register the VM.
|
||||
# No auth token or API URL exists, so neither this workflow nor the pinned
|
||||
# action can report hydration_failed. Fail explicitly; the CLI's bounded
|
||||
# readiness wait and receipt-aware cleanup remain the only safe recovery.
|
||||
- name: Require Testbox registration state
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [[ ! -d /tmp/.testbox ]]; then
|
||||
echo "::error::begin-testbox produced no registration state; hydration failure cannot be reported from this runner" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Confirm Testbox registration and mark setup ready
|
||||
if: success()
|
||||
env:
|
||||
EXPECTED_TESTBOX_ID: ${{ inputs.testbox_id }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
state=/tmp/.testbox
|
||||
rm -f /tmp/.testbox/cmux-tui-rust-setup-identity.json
|
||||
for required_file in testbox_id installation_model_id auth_token api_url runner_host runner_ssh_port adopted_run_id ssh_public_key; do
|
||||
test -s "$state/$required_file"
|
||||
done
|
||||
test -s "$state/working_directory"
|
||||
test "$(<"$state/testbox_id")" = "$EXPECTED_TESTBOX_ID"
|
||||
installation_model_id="$(<"$state/installation_model_id")"
|
||||
[[ "$installation_model_id" =~ ^[0-9]+$ ]]
|
||||
test -s "$state/ssh_public_key"
|
||||
# begin-testbox deliberately continues after degraded phone-home.
|
||||
# Validate the SSH handoff and install the identity marker before the
|
||||
# pinned run-testbox action publishes ready. That action remains the
|
||||
# sole readiness publisher, so a CLI run cannot race this marker.
|
||||
marker="$state/cmux-tui-rust-setup-identity.json"
|
||||
rm -f "$marker"
|
||||
install -m 600 testbox-benchmark/setup-identity.json "$marker"
|
||||
test -s "$marker"
|
||||
|
||||
# Keepalive is checked into this main-controlled workflow instead of
|
||||
# importing an upstream composite that performs an unbounded duplicate
|
||||
# ready curl. It reports
|
||||
# hydration_failed on any failed setup or readiness path and bounds every
|
||||
# phone-home request before entering its idle loop.
|
||||
- name: Run trusted Testbox keepalive
|
||||
if: always()
|
||||
env:
|
||||
JOB_STATUS: ${{ job.status }}
|
||||
shell: bash
|
||||
run: ./scripts/blacksmith-testbox-keepalive.sh
|
||||
@@ -74,3 +74,7 @@ artifacts/
|
||||
|
||||
# tmux verbose debug logs (tmux -v) that land in the cwd
|
||||
tmux-*.log
|
||||
|
||||
# Remote-only Blacksmith cmux-tui benchmark output
|
||||
/testbox-benchmark/
|
||||
/.cmux-scratch/
|
||||
|
||||
Executable
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ $# -lt 2 || ! "$1" =~ ^[0-9]+$ ]]; then
|
||||
echo "usage: $0 <timeout-seconds> <command> [args...]" >&2
|
||||
exit 64
|
||||
fi
|
||||
seconds="$1"
|
||||
shift
|
||||
|
||||
# Prefer GNU coreutils when present, while keeping the operator workflow
|
||||
# usable on macOS hosts that have neither `timeout` nor `gtimeout`.
|
||||
for candidate in gtimeout timeout; do
|
||||
if command -v "$candidate" >/dev/null && "$candidate" --version 2>&1 | grep -q 'GNU coreutils'; then
|
||||
exec "$candidate" --kill-after=5s "${seconds}s" "$@"
|
||||
fi
|
||||
done
|
||||
|
||||
exec python3 - "$seconds" "$@" <<'PY'
|
||||
import os
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
seconds = float(sys.argv[1])
|
||||
argv = sys.argv[2:]
|
||||
if not argv:
|
||||
raise SystemExit("missing command")
|
||||
process = subprocess.Popen(argv, start_new_session=True)
|
||||
|
||||
def terminate(_signum, _frame):
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGTERM)
|
||||
process.wait(timeout=5)
|
||||
except (ProcessLookupError, subprocess.TimeoutExpired):
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
process.wait()
|
||||
raise SystemExit(128 + _signum)
|
||||
|
||||
signal.signal(signal.SIGINT, terminate)
|
||||
signal.signal(signal.SIGTERM, terminate)
|
||||
try:
|
||||
raise SystemExit(process.wait(timeout=seconds))
|
||||
except subprocess.TimeoutExpired:
|
||||
terminate(signal.SIGTERM, None)
|
||||
PY
|
||||
Executable
+586
@@ -0,0 +1,586 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
export LC_ALL=C
|
||||
|
||||
# This helper is intentionally remote-only. The local benchmark plan invokes it
|
||||
# through `blacksmith testbox run`; the environment flag is only a first guard.
|
||||
# The Blacksmith kernel marker and /tmp/.testbox state below are the stronger
|
||||
# signals that this command is running in the prepared Testbox VM.
|
||||
if [[ "${CMUX_TESTBOX_REMOTE:-}" != "1" ]]; then
|
||||
echo "refusing to run outside a Blacksmith Testbox (set CMUX_TESTBOX_REMOTE=1 only in the remote command)" >&2
|
||||
exit 64
|
||||
fi
|
||||
if [[ ! -r /proc/cmdline ]] || ! grep -Eq '(^|[[:space:]])metadata_port=[^[:space:]]+' /proc/cmdline; then
|
||||
echo "refusing to run without the Blacksmith Testbox metadata marker" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
if [[ $# -ne 3 ]]; then
|
||||
echo "usage: CMUX_TESTBOX_REMOTE=1 CMUX_TESTBOX_ID=tbx_... $0 {first-clean|incremental-noop|changed-file} <source-sha> <ghostty-gitlink-sha>" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
stage="$1"
|
||||
expected_source_sha="$2"
|
||||
expected_ghostty_sha="$3"
|
||||
case "$stage" in
|
||||
first-clean|incremental-noop|changed-file) ;;
|
||||
*)
|
||||
echo "unsupported benchmark stage: $stage" >&2
|
||||
exit 64
|
||||
;;
|
||||
esac
|
||||
|
||||
for value_name in expected_source_sha expected_ghostty_sha; do
|
||||
value="${!value_name}"
|
||||
if [[ ! "$value" =~ ^[0-9a-f]{40}$ ]]; then
|
||||
echo "$value_name must be a lowercase 40-character commit SHA" >&2
|
||||
exit 64
|
||||
fi
|
||||
done
|
||||
|
||||
testbox_id="${CMUX_TESTBOX_ID:-}"
|
||||
if [[ ! "$testbox_id" =~ ^tbx_[A-Za-z0-9_-]+$ ]]; then
|
||||
echo "CMUX_TESTBOX_ID must identify the claimed Testbox" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
state_dir=/tmp/.testbox
|
||||
if [[ ! -d "$state_dir" || ! -s "$state_dir/auth_token" || ! -f "$state_dir/testbox_id" ]]; then
|
||||
echo "refusing to run without the Testbox state files" >&2
|
||||
exit 64
|
||||
fi
|
||||
state_testbox_id="$(tr -d '\r\n' <"$state_dir/testbox_id")"
|
||||
if [[ "$state_testbox_id" != "$testbox_id" ]]; then
|
||||
echo "Testbox state belongs to $state_testbox_id, expected $testbox_id" >&2
|
||||
exit 66
|
||||
fi
|
||||
|
||||
repo_root="$(git rev-parse --show-toplevel)"
|
||||
cd "$repo_root"
|
||||
ghostty_root="$repo_root/ghostty"
|
||||
if [[ ! -f cmux-tui/Cargo.toml || ! -f ghostty/build.zig.zon ]]; then
|
||||
echo "cmux-tui and its Ghostty source submodule must be initialized" >&2
|
||||
exit 65
|
||||
fi
|
||||
if [[ "$(git -C ghostty rev-parse --show-toplevel 2>/dev/null || true)" != "$ghostty_root" ]]; then
|
||||
echo "ghostty is not an initialized submodule checkout" >&2
|
||||
exit 65
|
||||
fi
|
||||
ghostty_entry="$(git ls-tree HEAD ghostty)"
|
||||
if [[ ! "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ]]; then
|
||||
echo "HEAD:ghostty is not a gitlink" >&2
|
||||
exit 65
|
||||
fi
|
||||
expected_tree_sha="$(git rev-parse "${expected_source_sha}^{tree}")"
|
||||
if ! command -v timeout >/dev/null; then
|
||||
echo "timeout is required for bounded remote builds" >&2
|
||||
exit 65
|
||||
fi
|
||||
setup_identity_path="$state_dir/cmux-tui-rust-setup-identity.json"
|
||||
if [[ ! -s "$setup_identity_path" ]]; then
|
||||
echo "refusing to run without a successful Testbox setup identity marker" >&2
|
||||
exit 65
|
||||
fi
|
||||
setup_run_id="$(tr -d '\r\n' <"$state_dir/adopted_run_id")"
|
||||
if [[ ! "$setup_run_id" =~ ^[0-9]+$ ]]; then
|
||||
echo "invalid Testbox setup workflow run ID" >&2
|
||||
exit 65
|
||||
fi
|
||||
# The broker hydrates main, while this stage benchmarks whatever revision the
|
||||
# operator synchronized onto the box. Those two commits are deliberately
|
||||
# allowed to differ, so the marker is checked for VM identity, runner class,
|
||||
# and toolchain completeness, not for source equality. The active-toolchain
|
||||
# gate further down is what still refuses a candidate whose pinned Rust or Zig
|
||||
# would leave the hydrated caches cold and the timings incomparable.
|
||||
verify_setup_identity() {
|
||||
python3 - "$setup_identity_path" "$testbox_id" "$setup_run_id" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
path, expected_testbox, expected_run_id = sys.argv[1:]
|
||||
try:
|
||||
record = json.loads(pathlib.Path(path).read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
raise SystemExit(f"invalid setup identity marker: {error}")
|
||||
source = record.get("source", {})
|
||||
testbox = record.get("testbox", {})
|
||||
runner = record.get("runner", {})
|
||||
toolchain = record.get("toolchain", {})
|
||||
errors = []
|
||||
if not re.fullmatch(r"[0-9a-f]{40}", str(source.get("commit_sha", ""))):
|
||||
errors.append("setup hydration commit is missing or malformed")
|
||||
if not re.fullmatch(r"[0-9a-f]{40}", str(source.get("tree_sha", ""))):
|
||||
errors.append("setup hydration tree is missing or malformed")
|
||||
if not re.fullmatch(r"[0-9a-f]{40}", str(source.get("ghostty_gitlink_sha", ""))):
|
||||
errors.append("setup hydration Ghostty gitlink is missing or malformed")
|
||||
if source.get("ghostty_head_sha") != source.get("ghostty_gitlink_sha"):
|
||||
errors.append("setup hydration Ghostty checkout does not match its own gitlink")
|
||||
if source.get("ref") != "refs/heads/main":
|
||||
errors.append(f"setup hydration ref {source.get('ref')!r} is not refs/heads/main")
|
||||
if testbox.get("id") != expected_testbox:
|
||||
errors.append("setup Testbox ID mismatch")
|
||||
if str(testbox.get("setup_workflow_run_id")) != expected_run_id:
|
||||
errors.append("setup workflow run ID mismatch")
|
||||
if runner.get("label") != "blacksmith-32vcpu-ubuntu-2404" or runner.get("arch") != "X64" or runner.get("cpu_count") != 32:
|
||||
errors.append("setup runner identity mismatch")
|
||||
if not toolchain.get("rust_toolchain") or not toolchain.get("rustc") or not toolchain.get("cargo") or not toolchain.get("zig"):
|
||||
errors.append("setup toolchain identity is incomplete")
|
||||
if errors:
|
||||
for error in errors:
|
||||
print(error, file=sys.stderr)
|
||||
raise SystemExit(66)
|
||||
PY
|
||||
}
|
||||
verify_setup_identity
|
||||
setup_rust_toolchain="$(python3 - "$setup_identity_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
record=json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
print(record["toolchain"]["rust_toolchain"])
|
||||
PY
|
||||
)"
|
||||
setup_rustc="$(python3 - "$setup_identity_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
record=json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
print(record["toolchain"]["rustc"])
|
||||
PY
|
||||
)"
|
||||
setup_cargo="$(python3 - "$setup_identity_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
record=json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
print(record["toolchain"]["cargo"])
|
||||
PY
|
||||
)"
|
||||
setup_zig="$(python3 - "$setup_identity_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
record=json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
print(record["toolchain"]["zig"])
|
||||
PY
|
||||
)"
|
||||
|
||||
hydrated_source_sha="$(python3 - "$setup_identity_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
record=json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
print(record["source"]["commit_sha"])
|
||||
PY
|
||||
)"
|
||||
hydrated_source_ref="$(python3 - "$setup_identity_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
record=json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
print(record["source"]["ref"])
|
||||
PY
|
||||
)"
|
||||
|
||||
benchmark_dir="$repo_root/testbox-benchmark"
|
||||
command -v flock >/dev/null || {
|
||||
echo "flock is required for serialized Testbox stages" >&2
|
||||
exit 65
|
||||
}
|
||||
mkdir -p "$benchmark_dir"
|
||||
# Testbox can acknowledge a run while its remote shell is still flushing
|
||||
# output. Serialize stages and hold the lock through all artifact writes so a
|
||||
# subsequent run cannot overwrite a prior stage's timing file.
|
||||
exec 9>"$benchmark_dir/.stage.lock"
|
||||
flock -x 9
|
||||
|
||||
log_path="$benchmark_dir/$stage.log"
|
||||
time_path="$benchmark_dir/$stage.time"
|
||||
json_path="$benchmark_dir/$stage.json"
|
||||
pre_identity_path="$benchmark_dir/.$stage.pre-identity.json"
|
||||
post_identity_path="$benchmark_dir/.$stage.post-identity.json"
|
||||
changed_file="cmux-tui/crates/cmux-tui/src/main.rs"
|
||||
changed_backup=""
|
||||
changed_backup_sha256=""
|
||||
changed_backup_size=""
|
||||
|
||||
restore_changed_file() {
|
||||
if [[ -n "$changed_backup" ]]; then
|
||||
if [[ ! -f "$changed_backup" ]]; then
|
||||
echo "changed-file backup disappeared: $changed_backup" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(wc -c <"$changed_backup")" != "$changed_backup_size" ||
|
||||
"$(sha256sum "$changed_backup" | cut -d ' ' -f 1)" != "$changed_backup_sha256" ]]; then
|
||||
echo "changed-file backup failed integrity verification" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! cp "$changed_backup" "$repo_root/$changed_file"; then
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(wc -c <"$repo_root/$changed_file")" != "$changed_backup_size" ||
|
||||
"$(sha256sum "$repo_root/$changed_file" | cut -d ' ' -f 1)" != "$changed_backup_sha256" ]]; then
|
||||
echo "restored source failed integrity verification" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! rm -f "$changed_backup"; then
|
||||
return 1
|
||||
fi
|
||||
changed_backup=""
|
||||
fi
|
||||
}
|
||||
|
||||
# Always restore the deliberately changed source, including when Cargo exits
|
||||
# non-zero. Do not let cleanup replace the build result unless restoration
|
||||
# itself fails.
|
||||
# shellcheck disable=SC2329 # invoked indirectly by the signal/EXIT traps
|
||||
finish_source() {
|
||||
local result=$?
|
||||
if [[ -n "$changed_backup" ]]; then
|
||||
if ! restore_changed_file; then
|
||||
echo "failed to restore $changed_file" >&2
|
||||
(( result == 0 )) && result=67
|
||||
fi
|
||||
fi
|
||||
exit "$result"
|
||||
}
|
||||
# Signals must remain failures even when they arrive after a successful command.
|
||||
# The EXIT trap performs the actual restoration exactly once.
|
||||
# shellcheck disable=SC2329 # invoked indirectly by signal traps
|
||||
interrupt_source() {
|
||||
local signal_name="$1"
|
||||
trap - TERM INT HUP
|
||||
case "$signal_name" in
|
||||
TERM) exit 143 ;;
|
||||
INT) exit 130 ;;
|
||||
HUP) exit 129 ;;
|
||||
esac
|
||||
}
|
||||
trap 'interrupt_source TERM' TERM
|
||||
trap 'interrupt_source INT' INT
|
||||
trap 'interrupt_source HUP' HUP
|
||||
trap finish_source EXIT
|
||||
|
||||
clean_status() {
|
||||
local top_status ghostty_status
|
||||
top_status="$(git status --porcelain=v1 --untracked-files=normal)"
|
||||
if [[ -n "$top_status" ]]; then
|
||||
printf '%s\n' "top-level source is dirty:" "$top_status" >&2
|
||||
return 1
|
||||
fi
|
||||
ghostty_status="$(git -C ghostty status --porcelain=v1 --untracked-files=normal)"
|
||||
if [[ -n "$ghostty_status" ]]; then
|
||||
printf '%s\n' "Ghostty submodule is dirty:" "$ghostty_status" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
capture_identity() {
|
||||
python3 - "$expected_source_sha" "$expected_tree_sha" "$expected_ghostty_sha" "$testbox_id" "$repo_root" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
expected_source_sha, expected_tree_sha, expected_ghostty_sha, testbox_id, repo_root = sys.argv[1:]
|
||||
repo = pathlib.Path(repo_root)
|
||||
ghostty = repo / "ghostty"
|
||||
|
||||
def run(command, cwd=repo):
|
||||
return subprocess.check_output(command, cwd=cwd, text=True, stderr=subprocess.STDOUT).strip()
|
||||
|
||||
def optional_file(path):
|
||||
try:
|
||||
return path.read_text(encoding="utf-8").strip()
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
def status(cwd=repo):
|
||||
return subprocess.check_output(
|
||||
["git", "status", "--porcelain=v1", "--untracked-files=normal"],
|
||||
cwd=cwd,
|
||||
text=True,
|
||||
).splitlines()
|
||||
|
||||
source_sha = run(["git", "rev-parse", "HEAD"])
|
||||
source_tree_sha = run(["git", "rev-parse", "HEAD^{tree}"])
|
||||
ghostty_gitlink_sha = run(["git", "rev-parse", "HEAD:ghostty"])
|
||||
ghostty_head_sha = run(["git", "-C", "ghostty", "rev-parse", "HEAD"])
|
||||
record = {
|
||||
"commit_sha": source_sha,
|
||||
"tree_sha": source_tree_sha,
|
||||
"expected_commit_sha": expected_source_sha,
|
||||
"expected_tree_sha": expected_tree_sha,
|
||||
"dirty_files": status(),
|
||||
"ghostty": {
|
||||
"gitlink_sha": ghostty_gitlink_sha,
|
||||
"expected_gitlink_sha": expected_ghostty_sha,
|
||||
"head_sha": ghostty_head_sha,
|
||||
"dirty_files": status(ghostty),
|
||||
},
|
||||
"testbox_id": testbox_id,
|
||||
"testbox_state": {
|
||||
"adopted_run_id": optional_file(pathlib.Path("/tmp/.testbox/adopted_run_id")),
|
||||
"runner_host": optional_file(pathlib.Path("/tmp/.testbox/runner_host")),
|
||||
"runner_ssh_port": optional_file(pathlib.Path("/tmp/.testbox/runner_ssh_port")),
|
||||
},
|
||||
}
|
||||
print(json.dumps(record, sort_keys=True))
|
||||
PY
|
||||
}
|
||||
|
||||
verify_identity() {
|
||||
local identity_path="$1"
|
||||
python3 - "$identity_path" "$expected_source_sha" "$expected_tree_sha" "$expected_ghostty_sha" "$testbox_id" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
path, expected_source_sha, expected_tree_sha, expected_ghostty_sha, expected_testbox_id = sys.argv[1:]
|
||||
record = json.loads(pathlib.Path(path).read_text(encoding="utf-8"))
|
||||
errors = []
|
||||
if record.get("commit_sha") != expected_source_sha:
|
||||
errors.append(f"source commit {record.get('commit_sha')} != {expected_source_sha}")
|
||||
if record.get("expected_commit_sha") != expected_source_sha:
|
||||
errors.append("source expectation was not recorded")
|
||||
if record.get("tree_sha") != expected_tree_sha:
|
||||
errors.append(f"source tree {record.get('tree_sha')} != {expected_tree_sha}")
|
||||
if record.get("expected_tree_sha") != expected_tree_sha:
|
||||
errors.append("source tree expectation was not recorded")
|
||||
if record.get("dirty_files"):
|
||||
errors.append("top-level source is dirty")
|
||||
ghostty = record.get("ghostty", {})
|
||||
if ghostty.get("gitlink_sha") != expected_ghostty_sha:
|
||||
errors.append(f"Ghostty gitlink {ghostty.get('gitlink_sha')} != {expected_ghostty_sha}")
|
||||
if ghostty.get("expected_gitlink_sha") != expected_ghostty_sha:
|
||||
errors.append("Ghostty expectation was not recorded")
|
||||
if ghostty.get("head_sha") != expected_ghostty_sha:
|
||||
errors.append(f"Ghostty checkout {ghostty.get('head_sha')} != {expected_ghostty_sha}")
|
||||
if ghostty.get("dirty_files"):
|
||||
errors.append("Ghostty submodule is dirty")
|
||||
if record.get("testbox_id") != expected_testbox_id:
|
||||
errors.append("Testbox identity mismatch")
|
||||
if errors:
|
||||
for error in errors:
|
||||
print(f"source guard: {error}", file=sys.stderr)
|
||||
raise SystemExit(66)
|
||||
PY
|
||||
}
|
||||
|
||||
# Verify the immutable source and submodule before every stage. The changed-file
|
||||
# stage is allowed to become dirty only after this check and must be clean again
|
||||
# before its JSON record is emitted.
|
||||
clean_status
|
||||
capture_identity >"$pre_identity_path"
|
||||
verify_identity "$pre_identity_path"
|
||||
|
||||
runner_label="blacksmith-32vcpu-ubuntu-2404"
|
||||
zig_bin="${CMUX_ZIG:-$(command -v zig)}"
|
||||
pushd cmux-tui >/dev/null
|
||||
rust_toolchain="$(rustup show active-toolchain)"
|
||||
rustc_version="$(rustc --version)"
|
||||
cargo_version="$(cargo --version)"
|
||||
popd >/dev/null
|
||||
zig_version="$("$zig_bin" version)"
|
||||
[[ "$rust_toolchain" == "$setup_rust_toolchain" && "$rustc_version" == "$setup_rustc" && "$cargo_version" == "$setup_cargo" && "$zig_version" == "$setup_zig" ]] || {
|
||||
echo "active Rust/Cargo/Zig toolchain differs from the setup identity marker" >&2
|
||||
exit 66
|
||||
}
|
||||
export ZIG="$zig_bin"
|
||||
rust_toolchain_file_sha256="$(sha256sum cmux-tui/rust-toolchain.toml | cut -d ' ' -f 1)"
|
||||
cargo_lock_sha256="$(sha256sum cmux-tui/Cargo.lock | cut -d ' ' -f 1)"
|
||||
ghostty_zon_sha256="$(sha256sum ghostty/build.zig.zon | cut -d ' ' -f 1)"
|
||||
|
||||
case "$stage" in
|
||||
first-clean)
|
||||
rm -rf "$repo_root/cmux-tui/target"
|
||||
;;
|
||||
changed-file)
|
||||
if [[ ! -f "$repo_root/$changed_file" ]]; then
|
||||
echo "changed-file target is missing: $changed_file" >&2
|
||||
exit 65
|
||||
fi
|
||||
backup_candidate="$(mktemp "${TMPDIR:-/tmp}/cmux-tui-testbox-source.XXXXXX")"
|
||||
if ! cp "$repo_root/$changed_file" "$backup_candidate"; then
|
||||
rm -f "$backup_candidate"
|
||||
echo "failed to create a source backup" >&2
|
||||
exit 67
|
||||
fi
|
||||
backup_sha256="$(sha256sum "$backup_candidate" | cut -d ' ' -f 1)"
|
||||
backup_size="$(wc -c <"$backup_candidate" | tr -d '[:space:]')"
|
||||
[[ "$backup_size" =~ ^[0-9]+$ && "$backup_size" -gt 0 ]] || {
|
||||
rm -f "$backup_candidate"
|
||||
echo "source backup is empty" >&2
|
||||
exit 67
|
||||
}
|
||||
changed_backup="$backup_candidate"
|
||||
changed_backup_sha256="$backup_sha256"
|
||||
changed_backup_size="$backup_size"
|
||||
printf '\n// Blacksmith Testbox changed-file timing marker.\n' >>"$repo_root/$changed_file"
|
||||
;;
|
||||
esac
|
||||
|
||||
start_epoch="$(python3 -c 'import time; print(time.time())')"
|
||||
rm -f "$time_path" "$log_path" "$json_path" "$post_identity_path"
|
||||
set +e
|
||||
(
|
||||
cd "$repo_root/cmux-tui"
|
||||
timeout --kill-after=30s 20m \
|
||||
/usr/bin/time -p -o "$time_path" cargo build -p cmux-tui --locked
|
||||
) >"$log_path" 2>&1
|
||||
build_status=$?
|
||||
set -e
|
||||
end_epoch="$(python3 -c 'import time; print(time.time())')"
|
||||
|
||||
restore_status=0
|
||||
if ! restore_changed_file; then
|
||||
echo "failed to restore $changed_file" >&2
|
||||
restore_status=67
|
||||
fi
|
||||
|
||||
post_identity_status=0
|
||||
if ! capture_identity >"$post_identity_path"; then
|
||||
echo "failed to capture post-stage source identity" >&2
|
||||
post_identity_status=66
|
||||
printf '{}\n' >"$post_identity_path"
|
||||
fi
|
||||
if (( post_identity_status == 0 )); then
|
||||
if ! verify_identity "$post_identity_path"; then
|
||||
post_identity_status=66
|
||||
fi
|
||||
fi
|
||||
|
||||
if (( restore_status != 0 )); then
|
||||
final_status="$restore_status"
|
||||
elif (( post_identity_status != 0 )); then
|
||||
final_status="$post_identity_status"
|
||||
else
|
||||
final_status="$build_status"
|
||||
fi
|
||||
|
||||
python3 - "$stage" "$start_epoch" "$end_epoch" "$build_status" "$final_status" "$time_path" "$pre_identity_path" "$post_identity_path" "$changed_file" "$expected_source_sha" "$expected_tree_sha" "$expected_ghostty_sha" "$testbox_id" "$runner_label" "$rust_toolchain" "$rustc_version" "$cargo_version" "$zig_bin" "$zig_version" "$rust_toolchain_file_sha256" "$cargo_lock_sha256" "$ghostty_zon_sha256" "$hydrated_source_ref" "$hydrated_source_sha" >"$json_path" <<'PY'
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import platform
|
||||
import sys
|
||||
|
||||
(
|
||||
stage,
|
||||
start,
|
||||
end,
|
||||
build_status,
|
||||
final_status,
|
||||
time_path,
|
||||
pre_identity_path,
|
||||
post_identity_path,
|
||||
changed_file,
|
||||
expected_source_sha,
|
||||
expected_tree_sha,
|
||||
expected_ghostty_sha,
|
||||
testbox_id,
|
||||
runner_label,
|
||||
rust_toolchain,
|
||||
rustc_version,
|
||||
cargo_version,
|
||||
zig_bin,
|
||||
zig_version,
|
||||
rust_toolchain_file_sha256,
|
||||
cargo_lock_sha256,
|
||||
ghostty_zon_sha256,
|
||||
hydrated_source_ref,
|
||||
hydrated_source_sha,
|
||||
) = sys.argv[1:]
|
||||
|
||||
def read_json(path):
|
||||
try:
|
||||
return json.loads(pathlib.Path(path).read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return {}
|
||||
|
||||
remote_time = {}
|
||||
try:
|
||||
for line in pathlib.Path(time_path).read_text(encoding="utf-8").splitlines():
|
||||
key, _, value = line.partition(" ")
|
||||
if key in {"real", "user", "sys"}:
|
||||
remote_time[f"time_{key}_seconds"] = float(value)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
pre = read_json(pre_identity_path)
|
||||
post = read_json(post_identity_path)
|
||||
record = {
|
||||
"schema": 3,
|
||||
"stage": stage,
|
||||
"command": "cargo build -p cmux-tui --locked",
|
||||
"build_exit_code": int(build_status),
|
||||
"exit_code": int(final_status),
|
||||
"ok": int(final_status) == 0,
|
||||
"started_at": dt.datetime.fromtimestamp(float(start), dt.timezone.utc).isoformat(),
|
||||
"finished_at": dt.datetime.fromtimestamp(float(end), dt.timezone.utc).isoformat(),
|
||||
"wall_seconds": round(float(end) - float(start), 3),
|
||||
"source": {
|
||||
"expected_commit_sha": expected_source_sha,
|
||||
"expected_tree_sha": expected_tree_sha,
|
||||
"before": pre,
|
||||
"after": post,
|
||||
"changed_file": changed_file if stage == "changed-file" else None,
|
||||
"restored": stage != "changed-file" or not post.get("dirty_files"),
|
||||
},
|
||||
"ghostty": {
|
||||
"expected_gitlink_sha": expected_ghostty_sha,
|
||||
"before_gitlink_sha": pre.get("ghostty", {}).get("gitlink_sha"),
|
||||
"before_head_sha": pre.get("ghostty", {}).get("head_sha"),
|
||||
"after_gitlink_sha": post.get("ghostty", {}).get("gitlink_sha"),
|
||||
"after_head_sha": post.get("ghostty", {}).get("head_sha"),
|
||||
},
|
||||
"testbox": {
|
||||
"id": testbox_id,
|
||||
"adopted_run_id": pre.get("testbox_state", {}).get("adopted_run_id"),
|
||||
},
|
||||
# What the broker warmed the caches from. It is main, and it is normally a
|
||||
# different commit from the benchmarked revision above.
|
||||
"hydration": {
|
||||
"ref": hydrated_source_ref,
|
||||
"commit_sha": hydrated_source_sha,
|
||||
"matches_benchmarked_source": hydrated_source_sha == expected_source_sha,
|
||||
},
|
||||
"runner": {
|
||||
"label": runner_label,
|
||||
"hostname": platform.node(),
|
||||
"arch": platform.machine(),
|
||||
"cpu_count": os.cpu_count(),
|
||||
"uname": " ".join(platform.uname()),
|
||||
"host_from_testbox_state": pre.get("testbox_state", {}).get("runner_host"),
|
||||
},
|
||||
"toolchain": {
|
||||
"rust_toolchain": rust_toolchain,
|
||||
"rustc": rustc_version,
|
||||
"cargo": cargo_version,
|
||||
"rust_toolchain_file_sha256": rust_toolchain_file_sha256,
|
||||
"cargo_lock_sha256": cargo_lock_sha256,
|
||||
"zig_path": zig_bin,
|
||||
"zig": zig_version,
|
||||
"ghostty_build_zig_zon_sha256": ghostty_zon_sha256,
|
||||
},
|
||||
**remote_time,
|
||||
}
|
||||
print(json.dumps(record, sort_keys=True))
|
||||
PY
|
||||
|
||||
cat "$log_path"
|
||||
printf '\n--- /usr/bin/time -p (%s) ---\n' "$stage"
|
||||
if [[ -f "$time_path" ]]; then
|
||||
cat "$time_path"
|
||||
else
|
||||
echo "time output unavailable" >&2
|
||||
fi
|
||||
printf '\n--- structured timing (%s) ---\n' "$stage"
|
||||
cat "$json_path"
|
||||
exit "$final_status"
|
||||
Executable
+368
@@ -0,0 +1,368 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ $# -ne 4 ]]; then
|
||||
echo "usage: $0 <testbox-id> <evidence-directory> <ownership-token> <PREVIEW|STOP:preview-sha>" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
testbox_id="$1"
|
||||
evidence_dir="$2"
|
||||
ownership_token="$3"
|
||||
operator_confirmation="$4"
|
||||
if [[ ! "$testbox_id" =~ ^tbx_[A-Za-z0-9_-]+$ ]]; then
|
||||
echo "invalid Testbox ID: $testbox_id" >&2
|
||||
exit 64
|
||||
fi
|
||||
if [[ ! "$ownership_token" =~ ^[0-9a-f]{32}$ ]]; then
|
||||
echo "ownership token must be a 32-character lowercase hex value" >&2
|
||||
exit 64
|
||||
fi
|
||||
if [[ "$operator_confirmation" != "PREVIEW" && ! "$operator_confirmation" =~ ^STOP:[0-9a-f]{64}$ ]]; then
|
||||
echo "confirmation must be PREVIEW or STOP:<64-character preview SHA>" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
mkdir -p "$evidence_dir"
|
||||
sha256_file() {
|
||||
if command -v sha256sum >/dev/null; then
|
||||
sha256sum "$1" | awk '{print $1}'
|
||||
elif command -v shasum >/dev/null; then
|
||||
shasum -a 256 "$1" | awk '{print $1}'
|
||||
else
|
||||
echo "sha256sum or shasum is required for cleanup preview hashing" >&2
|
||||
return 65
|
||||
fi
|
||||
}
|
||||
bounded_command() {
|
||||
scripts_dir="$(CDPATH='' cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
"$scripts_dir/blacksmith-bounded-command.sh" "$@"
|
||||
}
|
||||
receipt_path="$evidence_dir/testbox-receipt.json"
|
||||
if [[ ! -s "$receipt_path" ]]; then
|
||||
echo "refusing cleanup without the warmup ownership receipt: $receipt_path" >&2
|
||||
exit 65
|
||||
fi
|
||||
python3 - "$receipt_path" "$testbox_id" "$ownership_token" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
receipt_path, expected_id, expected_token = sys.argv[1:]
|
||||
try:
|
||||
receipt = json.loads(pathlib.Path(receipt_path).read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
raise SystemExit(f"invalid warmup ownership receipt: {error}")
|
||||
if receipt.get("testbox_id") != expected_id:
|
||||
raise SystemExit("cleanup ID does not match the warmup ownership receipt")
|
||||
if receipt.get("confirmation_token") != expected_token:
|
||||
raise SystemExit("ownership token does not match the warmup ownership receipt")
|
||||
for field in ("workflow", "job", "source_ref", "source_sha", "source_tree_sha", "ghostty_gitlink_sha"):
|
||||
if not receipt.get(field):
|
||||
raise SystemExit(f"warmup ownership receipt is missing {field}")
|
||||
PY
|
||||
receipt_workflow="$(python3 - "$receipt_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
print(json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))["workflow"])
|
||||
PY
|
||||
)"
|
||||
receipt_job="$(python3 - "$receipt_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
print(json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))["job"])
|
||||
PY
|
||||
)"
|
||||
receipt_ref="$(python3 - "$receipt_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
print(json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))["source_ref"])
|
||||
PY
|
||||
)"
|
||||
receipt_source_sha="$(python3 - "$receipt_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
print(json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))["source_sha"])
|
||||
PY
|
||||
)"
|
||||
receipt_source_tree_sha="$(python3 - "$receipt_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
print(json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))["source_tree_sha"])
|
||||
PY
|
||||
)"
|
||||
receipt_ghostty_sha="$(python3 - "$receipt_path" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
print(json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))["ghostty_gitlink_sha"])
|
||||
PY
|
||||
)"
|
||||
|
||||
# Parse either the table emitted by `list/status --id` or a summary response.
|
||||
# The parser validates context only for a row containing this exact ID. It does
|
||||
# not rely on fixed whitespace columns, because queued rows may have an empty IP.
|
||||
parse_cli_output() {
|
||||
local log_path="$1"
|
||||
python3 - "$log_path" "$testbox_id" "$receipt_workflow" "$receipt_job" "$receipt_ref" <<'PY'
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
path, expected_id, expected_workflow, expected_job, expected_ref = sys.argv[1:]
|
||||
text = pathlib.Path(path).read_text(encoding="utf-8", errors="replace")
|
||||
for line in text.splitlines():
|
||||
fields = line.split()
|
||||
if not fields or fields[0] != expected_id:
|
||||
continue
|
||||
if len(fields) < 2:
|
||||
raise SystemExit(66)
|
||||
status = fields[1].lower()
|
||||
# Blacksmith CLI releases have emitted both ID/STATUS/REPO/WORKFLOW/CREATED
|
||||
# and ID/STATUS/IP/WORKFLOW/JOB/REF/... schemas. Require the exact workflow
|
||||
# in either schema; when job/ref columns exist, require those too.
|
||||
try:
|
||||
workflow_index = fields.index(expected_workflow, 2)
|
||||
except ValueError:
|
||||
raise SystemExit(66)
|
||||
trailing = fields[workflow_index + 1:]
|
||||
# Known schemas either end after CREATED (no job/ref columns) or expose
|
||||
# JOB and REF immediately after WORKFLOW. If either expected field appears,
|
||||
# require the exact pair in the exact order; reject all ambiguous contexts.
|
||||
if len(trailing) >= 2 and trailing[:2] == [expected_job, expected_ref]:
|
||||
pass
|
||||
elif expected_job in trailing or expected_ref in trailing:
|
||||
raise SystemExit(66)
|
||||
elif len(trailing) not in (1, 2):
|
||||
raise SystemExit(66)
|
||||
print(status)
|
||||
raise SystemExit(0)
|
||||
|
||||
# Some CLI versions use a summary such as `[tbx_...] Status: ready`.
|
||||
if re.search(rf"\b{re.escape(expected_id)}\b", text):
|
||||
match = re.search(r"\bstatus\s*:?\s*([A-Za-z_]+)", text, re.IGNORECASE)
|
||||
if match:
|
||||
print(match.group(1).lower())
|
||||
raise SystemExit(0)
|
||||
raise SystemExit(3)
|
||||
PY
|
||||
}
|
||||
|
||||
is_terminal() {
|
||||
case "$1" in
|
||||
completed|stopped|cancelled|failed|terminated|hydration_failed) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
is_active() {
|
||||
case "$1" in
|
||||
ready|running|hydrating|in_progress|queued) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
is_known_absence() {
|
||||
grep -Eiq '(not found|already[[:space:]]+(stopped|completed)|hydration_failed|HTTP[[:space:]]+404|status[[:space:]]+code[[:space:]]+404|HTTP[[:space:]]+409|status[[:space:]]+code[[:space:]]+409)' "$1"
|
||||
}
|
||||
|
||||
inventory_log="$evidence_dir/list-before-stop.log"
|
||||
set +e
|
||||
bounded_command 20 blacksmith testbox list --all >"$inventory_log" 2>&1
|
||||
inventory_status=$?
|
||||
set -e
|
||||
if (( inventory_status != 0 )); then
|
||||
echo "failed to capture the Testbox inventory before cleanup; refusing stop" >&2
|
||||
exit "$inventory_status"
|
||||
fi
|
||||
|
||||
inventory_row_present=0
|
||||
set +e
|
||||
parse_cli_output "$inventory_log" >/dev/null
|
||||
inventory_parse_status=$?
|
||||
set -e
|
||||
case "$inventory_parse_status" in
|
||||
0) inventory_row_present=1 ;;
|
||||
3) : ;;
|
||||
66) echo "inventory ownership context differs from the warmup receipt; refusing cleanup" >&2; exit 66 ;;
|
||||
*) echo "could not parse the Testbox inventory; refusing cleanup" >&2; exit "$inventory_parse_status" ;;
|
||||
esac
|
||||
|
||||
status_log="$evidence_dir/status-before-stop.log"
|
||||
set +e
|
||||
bounded_command 20 blacksmith testbox status --id "$testbox_id" >"$status_log" 2>&1
|
||||
status_command_status=$?
|
||||
set -e
|
||||
status_value=""
|
||||
status_absent=0
|
||||
if (( status_command_status == 0 )); then
|
||||
set +e
|
||||
status_value="$(parse_cli_output "$status_log")"
|
||||
status_parse_status=$?
|
||||
set -e
|
||||
case "$status_parse_status" in
|
||||
0) ;;
|
||||
3) echo "status omitted the owned Testbox $testbox_id; refusing cleanup" >&2; exit 66 ;;
|
||||
66) echo "status ownership context differs from the warmup receipt; refusing cleanup" >&2; exit 66 ;;
|
||||
*) echo "could not parse status for owned Testbox $testbox_id; refusing cleanup" >&2; exit "$status_parse_status" ;;
|
||||
esac
|
||||
elif is_known_absence "$status_log"; then
|
||||
status_absent=1
|
||||
else
|
||||
echo "failed to inspect owned Testbox $testbox_id before cleanup; refusing stop" >&2
|
||||
exit "$status_command_status"
|
||||
fi
|
||||
|
||||
if (( status_absent == 0 )) && is_active "$status_value" && (( inventory_row_present == 0 )); then
|
||||
echo "owned Testbox is active but absent from the inventory; refusing cleanup" >&2
|
||||
exit 66
|
||||
fi
|
||||
if (( status_absent == 0 )) && ! is_active "$status_value" && ! is_terminal "$status_value"; then
|
||||
echo "unknown status for owned Testbox $testbox_id; refusing cleanup" >&2
|
||||
exit 66
|
||||
fi
|
||||
|
||||
preview_path="$evidence_dir/cleanup-preview.json"
|
||||
python3 - "$preview_path" "$testbox_id" "${status_value:-absent}" "$inventory_row_present" "$receipt_workflow" "$receipt_job" "$receipt_ref" "$receipt_source_sha" "$receipt_source_tree_sha" "$receipt_ghostty_sha" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
(path, testbox_id, status, inventory_present, workflow, job, ref,
|
||||
source_sha, source_tree_sha, ghostty_sha) = sys.argv[1:]
|
||||
payload = {
|
||||
"schema": 1,
|
||||
"testbox_id": testbox_id,
|
||||
"status": status,
|
||||
"inventory_row_present": bool(int(inventory_present)),
|
||||
"workflow": workflow,
|
||||
"job": job,
|
||||
"source_ref": ref,
|
||||
"source_sha": source_sha,
|
||||
"source_tree_sha": source_tree_sha,
|
||||
"ghostty_gitlink_sha": ghostty_sha,
|
||||
}
|
||||
out = pathlib.Path(path)
|
||||
out.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
out.chmod(0o600)
|
||||
PY
|
||||
preview_sha="$(sha256_file "$preview_path" | awk '{print $1}')"
|
||||
printf 'Testbox cleanup preview: id=%s status=%s inventory_row=%s workflow=%s job=%s ref=%s\n' \
|
||||
"$testbox_id" "${status_value:-absent}" "$inventory_row_present" "$receipt_workflow" "$receipt_job" "$receipt_ref"
|
||||
printf 'Preview SHA: %s\n' "$preview_sha"
|
||||
if [[ "$operator_confirmation" == "PREVIEW" ]]; then
|
||||
echo "Review the preview, then rerun with the same token and STOP:$preview_sha to authorize stop." >&2
|
||||
exit 75
|
||||
fi
|
||||
expected_preview_sha="${operator_confirmation#STOP:}"
|
||||
if [[ "$operator_confirmation" != "PREVIEW" && "$expected_preview_sha" != "$preview_sha" ]]; then
|
||||
echo "current cleanup preview differs from the supplied confirmation; refusing stop" >&2
|
||||
exit 67
|
||||
fi
|
||||
|
||||
stop_log="$evidence_dir/stop.log"
|
||||
list_log="$evidence_dir/list-after-stop.log"
|
||||
cleanup_status=0
|
||||
poll_deadline=$((SECONDS + 120))
|
||||
poll_attempt=0
|
||||
if (( status_absent == 1 )) || is_terminal "$status_value"; then
|
||||
printf 'Testbox %s is already terminal or absent; no stop request needed\n' "$testbox_id" >"$stop_log"
|
||||
else
|
||||
set +e
|
||||
bounded_command 20 blacksmith testbox stop --id "$testbox_id" >"$stop_log" 2>&1
|
||||
stop_status=$?
|
||||
set -e
|
||||
if (( stop_status != 0 )); then
|
||||
if is_known_absence "$stop_log"; then
|
||||
printf 'stop reached a known terminal or absent state for %s; continuing\n' "$testbox_id" >&2
|
||||
else
|
||||
echo "failed to stop Testbox $testbox_id; see $stop_log" >&2
|
||||
cleanup_status=$stop_status
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Poll the ID-specific endpoint until cancellation propagates. Never treat a
|
||||
# different row in the global inventory as proof that this ID is terminal.
|
||||
while :; do
|
||||
poll_attempt=$((poll_attempt + 1))
|
||||
: >"$status_log"
|
||||
set +e
|
||||
bounded_command 20 blacksmith testbox status --id "$testbox_id" >"$status_log" 2>&1
|
||||
status_command_status=$?
|
||||
set -e
|
||||
if (( status_command_status == 0 )); then
|
||||
set +e
|
||||
status_value="$(parse_cli_output "$status_log")"
|
||||
status_parse_status=$?
|
||||
set -e
|
||||
if (( status_parse_status != 0 )); then
|
||||
echo "could not parse post-stop status for $testbox_id; see $status_log" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=66
|
||||
break
|
||||
fi
|
||||
if is_terminal "$status_value"; then
|
||||
break
|
||||
fi
|
||||
if ! is_active "$status_value"; then
|
||||
echo "unknown post-stop status for $testbox_id; see $status_log" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=66
|
||||
break
|
||||
fi
|
||||
elif is_known_absence "$status_log"; then
|
||||
break
|
||||
else
|
||||
echo "failed to inspect Testbox $testbox_id after cleanup; see $status_log" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=$status_command_status
|
||||
break
|
||||
fi
|
||||
if (( SECONDS >= poll_deadline )); then
|
||||
echo "Testbox $testbox_id is still active after bounded cleanup polling" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=1
|
||||
break
|
||||
fi
|
||||
sleep_seconds=$((poll_attempt < 6 ? poll_attempt * 2 : 10))
|
||||
sleep "$sleep_seconds"
|
||||
done
|
||||
|
||||
set +e
|
||||
bounded_command 20 blacksmith testbox list --all >"$list_log" 2>&1
|
||||
list_status=$?
|
||||
set -e
|
||||
if (( list_status != 0 )); then
|
||||
echo "failed to list Testboxes after stopping $testbox_id; see $list_log" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=$list_status
|
||||
else
|
||||
set +e
|
||||
listed_status="$(parse_cli_output "$list_log")"
|
||||
listed_parse_status=$?
|
||||
set -e
|
||||
case "$listed_parse_status" in
|
||||
0)
|
||||
if is_active "$listed_status"; then
|
||||
echo "Testbox $testbox_id is still active after cleanup; see $list_log" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=1
|
||||
elif ! is_terminal "$listed_status"; then
|
||||
echo "unknown status for Testbox $testbox_id in inventory: $listed_status" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=66
|
||||
fi
|
||||
;;
|
||||
3) ;;
|
||||
66)
|
||||
echo "Testbox $testbox_id ownership changed in final inventory; see $list_log" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=66
|
||||
;;
|
||||
*)
|
||||
echo "could not parse final Testbox inventory; see $list_log" >&2
|
||||
(( cleanup_status == 0 )) && cleanup_status=$listed_parse_status
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
|
||||
if (( cleanup_status != 0 )); then
|
||||
exit "$cleanup_status"
|
||||
fi
|
||||
printf 'verified Testbox %s is no longer active\n' "$testbox_id"
|
||||
Executable
+108
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# This is the trusted-main replacement for the upstream keepalive composite.
|
||||
# It deliberately reads the Testbox token only after GitHub has evaluated the
|
||||
# protected environment and the workflow's main/repository guard.
|
||||
state=/tmp/.testbox
|
||||
job_status="${JOB_STATUS:-failure}"
|
||||
|
||||
if [[ ! -d "$state" ]]; then
|
||||
if [[ "$job_status" == "success" ]]; then
|
||||
echo "Testbox validation passed, but no registration state was returned" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Testbox registration state is absent after a failed setup; no phone-home is possible" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
for required_file in testbox_id installation_model_id auth_token api_url runner_host runner_ssh_port adopted_run_id working_directory; do
|
||||
test -s "$state/$required_file" || {
|
||||
echo "missing Testbox state file: $state/$required_file" >&2
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
testbox_id="$(<"$state/testbox_id")"
|
||||
installation_model_id="$(<"$state/installation_model_id")"
|
||||
auth_token="$(<"$state/auth_token")"
|
||||
api_url="$(<"$state/api_url")"
|
||||
runner_host="$(<"$state/runner_host")"
|
||||
runner_ssh_port="$(<"$state/runner_ssh_port")"
|
||||
working_directory="$(<"$state/working_directory")"
|
||||
adopted_run_id="$(<"$state/adopted_run_id")"
|
||||
[[ "$testbox_id" =~ ^tbx_[A-Za-z0-9_-]+$ ]]
|
||||
[[ "$installation_model_id" =~ ^[0-9]+$ ]]
|
||||
|
||||
phone_home() {
|
||||
local status="$1"
|
||||
local payload
|
||||
payload="$(jq -n \
|
||||
--arg testbox_id "$testbox_id" \
|
||||
--arg runner_host "$runner_host" \
|
||||
--arg runner_ssh_port "$runner_ssh_port" \
|
||||
--arg working_directory "$working_directory" \
|
||||
--arg adopted_run_id "$adopted_run_id" \
|
||||
--arg status "$status" \
|
||||
--argjson installation_model_id "$installation_model_id" \
|
||||
'{testbox_id: $testbox_id, installation_model_id: $installation_model_id, status: $status, ip_address: $runner_host, ssh_port: $runner_ssh_port, working_directory: $working_directory, adopted_run_id: $adopted_run_id, metadata: {}}')"
|
||||
curl --fail --silent --show-error --connect-timeout 2 --max-time 10 \
|
||||
-X POST "$api_url/api/testbox/phone-home" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H "Authorization: Bearer $auth_token" \
|
||||
--data "$payload" >/dev/null
|
||||
}
|
||||
|
||||
phone_home_with_retry() {
|
||||
local status="$1"
|
||||
local attempt
|
||||
for attempt in 1 2 3 4 5; do
|
||||
if phone_home "$status"; then
|
||||
return 0
|
||||
fi
|
||||
if (( attempt < 5 )); then
|
||||
sleep $((attempt * 2))
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
if [[ "$job_status" != "success" ]]; then
|
||||
if ! phone_home_with_retry hydration_failed; then
|
||||
echo "warning: could not report hydration_failed" >&2
|
||||
fi
|
||||
echo "Testbox hydration failed; no ready state was published" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! phone_home_with_retry ready; then
|
||||
echo "ready phone-home failed after bounded retries" >&2
|
||||
phone_home_with_retry hydration_failed || echo "warning: could not report hydration_failed" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf 'Testbox ready: %s (%s)\n' "$testbox_id" "$runner_host"
|
||||
idle_timeout_minutes="10"
|
||||
if [[ -s "$state/idle_timeout" ]]; then
|
||||
idle_timeout_minutes="$(cat "$state/idle_timeout")"
|
||||
fi
|
||||
[[ "$idle_timeout_minutes" =~ ^[0-9]+$ ]] || idle_timeout_minutes=10
|
||||
last_activity="$(date +%s)"
|
||||
idle_timeout_seconds=$((idle_timeout_minutes * 60))
|
||||
|
||||
while :; do
|
||||
sleep 30
|
||||
now="$(date +%s)"
|
||||
if ss -tnp 2>/dev/null | grep -Eq ":${runner_ssh_port}([^0-9]|$)"; then
|
||||
last_activity="$now"
|
||||
elif [[ -f "$HOME/.testbox-last-activity" ]]; then
|
||||
marker_mtime="$(stat -c %Y "$HOME/.testbox-last-activity" 2>/dev/null || stat -f %m "$HOME/.testbox-last-activity")"
|
||||
if [[ "$marker_mtime" -gt "$last_activity" ]]; then
|
||||
last_activity="$marker_mtime"
|
||||
fi
|
||||
fi
|
||||
if (( now - last_activity >= idle_timeout_seconds )); then
|
||||
phone_home_with_retry completed || echo "warning: could not report completed" >&2
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
@@ -0,0 +1,380 @@
|
||||
---
|
||||
name: blacksmith-testbox
|
||||
description: >
|
||||
Provision and reuse a trusted Blacksmith Testbox for cmux-tui Rust builds,
|
||||
capture remote timings, download raw evidence, and clean up safely. Never run
|
||||
cargo, rustc, or Zig builds on the local Mac.
|
||||
---
|
||||
|
||||
# cmux-tui Blacksmith Testbox
|
||||
|
||||
This lane is Linux-only. It uses
|
||||
`.github/workflows/cmux-tui-testbox-warmup.yml`, job
|
||||
`cmux-tui-rust`, on `blacksmith-32vcpu-ubuntu-2404`. The workflow is a
|
||||
setup-only entrypoint for a reusable Testbox. It refuses every ref except
|
||||
`refs/heads/main`, checks out and verifies that commit, initializes the
|
||||
`ghostty` source submodule, installs Linux C/LLVM headers, installs the
|
||||
repository-pinned Zig and Rust toolchains, fetches Zig and Cargo dependencies,
|
||||
records runner/toolchain/Ghostty identity in JSON, and then hands control back
|
||||
to Testbox. It does not run Rust tests or Rust compilation during warmup.
|
||||
`zig build --fetch` only hydrates Zig packages and exits before compilation.
|
||||
|
||||
The repository's single Rust toolchain source is
|
||||
`cmux-tui/rust-toolchain.toml`. The workflow invokes
|
||||
`./.github/actions/setup-cmux-tui-rust`, so a workflow-specific Rust version
|
||||
must never be added.
|
||||
|
||||
## Hard safety and trust boundary
|
||||
|
||||
This is a main-controlled broker lane. `useblacksmith/begin-testbox` writes
|
||||
`/tmp/.testbox/auth_token` into the job, and `permissions: contents: read` does
|
||||
not stop any later step from reading it. The lane therefore holds one rule
|
||||
above all others: **nothing a candidate branch can edit ever runs inside that
|
||||
job.**
|
||||
|
||||
`blacksmith testbox warmup` resolves the workflow file and the hydrated source
|
||||
from the same `--ref`, so the two cannot be separated. The lane resolves that
|
||||
by hydrating `main` only. The first step fails any ref other than
|
||||
`refs/heads/main`, and every guard, the pinned `begin-testbox` SHA, and the
|
||||
keepalive all come from `main`. A candidate revision never becomes the workflow
|
||||
definition, and its `build.zig`, `rust-toolchain.toml`, and composite actions
|
||||
never execute in the token-bearing job.
|
||||
|
||||
A candidate reaches the Testbox afterwards, through `blacksmith testbox run`,
|
||||
which synchronizes a maintainer's local worktree onto the warm VM. That command
|
||||
runs as an authenticated Blacksmith organization member who could already read
|
||||
the box, so it moves no trust boundary. It does mean the box holds one
|
||||
operator's revision at a time: a Testbox ID belongs to one worktree and one
|
||||
operator.
|
||||
|
||||
Before using the lane, a repository administrator must create the
|
||||
`blacksmith-testbox-trusted` GitHub environment, configure required reviewers
|
||||
(or an equivalent manual approval rule), disable administrator bypass, leave
|
||||
the environment secret set empty, and set its deployment branch rule to exactly
|
||||
`main` with no wildcard and no fork rule. GitHub evaluates that approval and
|
||||
branch rule before the job's first step, so both precede `begin-testbox`. The
|
||||
approval is what authorizes spending a 32 vCPU box, because the code path is
|
||||
already fixed by `main`.
|
||||
|
||||
This lane needs no `BLACKSMITH_TESTBOX_REVIEWED_REF` or
|
||||
`BLACKSMITH_TESTBOX_REVIEWED_SHA` environment variable. Those pins existed to
|
||||
make a candidate-controlled workflow safe. Delete them if they are still
|
||||
configured; the broker's `refs/heads/main` guard replaces them, and it cannot
|
||||
be edited from a pull request.
|
||||
|
||||
If the environment is deleted, renamed, or loses its exact `main` branch rule,
|
||||
disable the lane and stop rather than changing the workflow to proceed. The
|
||||
workflow cannot manufacture those controls, so configuration drift makes the
|
||||
lane unavailable rather than safe.
|
||||
|
||||
The helper retains the `CMUX_TESTBOX_REMOTE=1` guard for accidental local
|
||||
launches, and additionally requires the Blacksmith VM kernel metadata marker
|
||||
and matching `/tmp/.testbox` state. The environment flag remains caller
|
||||
controlled and is not an authentication mechanism.
|
||||
|
||||
* Never run `cargo`, `rustc`, `rustup`, `zig build`, or another Rust/Zig build
|
||||
command on Lawrence's Mac. This includes local fallback builds and local
|
||||
test commands.
|
||||
* Run every Blacksmith CLI command from the root of the intended isolated
|
||||
worktree. The CLI synchronizes that directory with `rsync --delete`; it can
|
||||
delete remote files that are not represented locally.
|
||||
* Put remote build commands inside `blacksmith testbox run`. The benchmark
|
||||
helper also requires the Testbox VM guard, so an accidental local launch
|
||||
exits before invoking a compiler.
|
||||
* Use this Testbox only for Linux-compatible cmux-tui work. Use hosted macOS
|
||||
workflows for Swift, Xcode, XCTest, GUI, and app-host verification.
|
||||
|
||||
## Authentication and CLI installation
|
||||
|
||||
Check authentication without printing credentials:
|
||||
|
||||
```bash
|
||||
blacksmith auth whoami
|
||||
blacksmith --version
|
||||
```
|
||||
|
||||
Use the repository or organization-approved, pinned Blacksmith CLI artifact or
|
||||
package-manager version. Record its version in the evidence. Do not install a
|
||||
mutable remote script with `curl ... | sh`; if the approved pinned artifact is
|
||||
not available, stop and ask the tooling owner rather than weakening this lane.
|
||||
The repository does not currently pin a checksum-verified CLI artifact, so CLI
|
||||
provenance is a trusted-lane operational limitation: do not silently upgrade or
|
||||
substitute a version, and retain `blacksmith --version` with each evidence set.
|
||||
|
||||
## Exact source contract
|
||||
|
||||
Two commits matter and they are normally different. The **hydration commit** is
|
||||
`main`, and it is what the broker warmed the Cargo registry, Zig cache, and
|
||||
toolchains from. The **benchmarked commit** is your local HEAD, and it reaches
|
||||
the box through `blacksmith testbox run`. The stage helper records both and
|
||||
sets `hydration.matches_benchmarked_source` in each stage JSON.
|
||||
|
||||
Do not pass a commit SHA to `blacksmith testbox warmup --ref`. Blacksmith and
|
||||
GitHub reject a raw SHA with HTTP 422 (`No ref found`), and this lane accepts
|
||||
only `main` anyway.
|
||||
|
||||
Your local HEAD must be committed and clean, because every remote stage
|
||||
verifies HEAD, its tree, the `ghostty` gitlink, the initialized Ghostty HEAD,
|
||||
and clean status before it builds. Pushing that commit is not required for the
|
||||
build, since the sync carries the objects, but push it anyway so the evidence
|
||||
names a fetchable revision. Initialize the public Ghostty submodule once, then
|
||||
run this preflight from the clean worktree root:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
git submodule update --init ghostty
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
SOURCE_REF="$(git symbolic-ref --short HEAD)"
|
||||
if [[ ! "$SOURCE_REF" =~ ^[A-Za-z0-9._/-]+$ || "$SOURCE_REF" == *..* || "$SOURCE_REF" == */ || "$SOURCE_REF" == *//* ]]; then
|
||||
echo "HEAD must name a plain branch" >&2
|
||||
exit 1
|
||||
fi
|
||||
SOURCE_SHA="$(git rev-parse HEAD)"
|
||||
ghostty_entry="$(git ls-tree HEAD ghostty)"
|
||||
[[ "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ]] || {
|
||||
echo "HEAD:ghostty is not a gitlink" >&2
|
||||
exit 1
|
||||
}
|
||||
GHOSTTY_SHA="$(git rev-parse HEAD:ghostty)"
|
||||
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
|
||||
[[ "$GHOSTTY_SHA" =~ ^[0-9a-f]{40}$ ]]
|
||||
[[ "$(git -C ghostty rev-parse HEAD)" == "$GHOSTTY_SHA" ]]
|
||||
[[ -z "$(git status --porcelain=v1 --untracked-files=normal)" ]]
|
||||
[[ -z "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]]
|
||||
BROKER_SHA="$(git ls-remote --exit-code --heads https://github.com/manaflow-ai/cmux.git refs/heads/main | awk 'NR == 1 { print $1 }')"
|
||||
printf 'source_ref=%s\nsource_sha=%s\nghostty_gitlink_sha=%s\nbroker_main_sha=%s\n' \
|
||||
"$SOURCE_REF" "$SOURCE_SHA" "$GHOSTTY_SHA" "$BROKER_SHA"
|
||||
```
|
||||
|
||||
`main` can move after this check, which only changes cache warmth, never
|
||||
correctness: the stage helper reads the hydration commit back out of the setup
|
||||
marker and records it. A moved or dirty benchmarked checkout still fails
|
||||
closed.
|
||||
|
||||
The hydrated toolchain is a hard gate. If your branch pins a different Rust or
|
||||
Zig than `main`, the stage helper exits 66 rather than reporting a timing
|
||||
against caches it did not warm. Rebase onto `main` or warm a fresh box from a
|
||||
`main` that carries the same pins.
|
||||
|
||||
## Warmup and identity capture
|
||||
|
||||
Warm from `main`. The broker refuses every other ref, and your candidate does
|
||||
not belong here; it arrives later through the sync:
|
||||
|
||||
```bash
|
||||
WORKFLOW=.github/workflows/cmux-tui-testbox-warmup.yml
|
||||
JOB=cmux-tui-rust
|
||||
blacksmith testbox warmup "$WORKFLOW" \
|
||||
--ref main \
|
||||
--job "$JOB" \
|
||||
--idle-timeout 30
|
||||
```
|
||||
|
||||
Save the returned `tbx_...` ID. One ID belongs to one worktree and one
|
||||
operator. The workflow concurrency group keys every setup request by Testbox
|
||||
ID, including requests for different source SHAs. Source identity is validated
|
||||
separately, and each remote stage holds `testbox-benchmark/.stage.lock` through
|
||||
its build and artifact writes. Blacksmith's CLI exposes no pre-rsync lease, so
|
||||
that remote lock cannot serialize the CLI's initial sync. One Testbox ID must
|
||||
therefore have one owning worktree/operator; do not issue concurrent `run` or
|
||||
download commands from independent clients. This is an explicit trusted-lane
|
||||
limitation, not a claim of multi-client Testbox isolation.
|
||||
|
||||
Wait for setup and capture the exact run identity:
|
||||
|
||||
```bash
|
||||
blacksmith testbox status --id "$TBX" --wait --wait-timeout 15m
|
||||
```
|
||||
|
||||
The setup job's `setup-identity.json` artifact and each stage helper's
|
||||
pre-build JSON record are the identity transcripts. The helper verifies the
|
||||
Testbox VM marker, claimed Testbox ID, the hydration marker's own consistency,
|
||||
the runner class, and then the synchronized source commit/tree, Ghostty
|
||||
gitlink/checkout, and clean status before it invokes Cargo, repeating the
|
||||
source checks after the build. Keep the setup artifact URL or download it into `$OUT`;
|
||||
it records runner/toolchain/Ghostty identity independently of the stage helper.
|
||||
The setup JSON contains the workflow run, the hydrated `main` ref/SHA/tree,
|
||||
Ghostty gitlink and checkout SHA, runner label/architecture/CPU identity, and pinned
|
||||
Rust/Zig/toolchain-file metadata. A successful setup also copies that JSON to
|
||||
`/tmp/.testbox/cmux-tui-rust-setup-identity.json`; the stage helper refuses a
|
||||
missing or mismatched marker, so a failed hydration cannot be benchmarked.
|
||||
Never print or download `/tmp/.testbox/auth_token`.
|
||||
|
||||
## Remote benchmark stages
|
||||
|
||||
The detailed, receipt-producing orchestration in `benchmark.md` is the required
|
||||
entry point for a complete benchmark. It creates the unique `OUT_ROOT`, receipt,
|
||||
cleanup token, setup artifact capture, and cleanup preview state. Do not copy
|
||||
only this stage loop into an ad hoc shell without those prerequisites.
|
||||
|
||||
Before each stage, recompute `SOURCE_SHA` and `GHOSTTY_SHA` and repeat the
|
||||
clean pushed-branch preflight, including the protected reviewed ref/SHA pins.
|
||||
Pass the expected values as validated arguments;
|
||||
the helper does not trust the remote checkout or a caller-supplied expected SHA
|
||||
without comparing it to Git metadata:
|
||||
|
||||
```bash
|
||||
# Run this orchestration block in Bash, not an interactive zsh session.
|
||||
run_stage() {
|
||||
local stage="$1"
|
||||
local run_status download_status=0
|
||||
set +e
|
||||
printf -v remote_command \
|
||||
'CMUX_TESTBOX_REMOTE=1 CMUX_TESTBOX_ID=%q %q %q %q %q' \
|
||||
"$TBX" ./scripts/blacksmith-cmux-tui-testbox-stage.sh \
|
||||
"$stage" "$SOURCE_SHA" "$GHOSTTY_SHA"
|
||||
./scripts/blacksmith-bounded-command.sh 1500 \
|
||||
blacksmith testbox run --id "$TBX" --debug \
|
||||
"$remote_command" >"$OUT/$stage.run.log" 2>&1
|
||||
run_status=$?
|
||||
set -e
|
||||
cat "$OUT/$stage.run.log"
|
||||
|
||||
# Download immediately. Blacksmith's next rsync may delete or replace remote
|
||||
# files, so a one-time download after all stages is insufficient.
|
||||
: >"$OUT/$stage.download.log"
|
||||
for suffix in json time log; do
|
||||
if ! ./scripts/blacksmith-bounded-command.sh 120 \
|
||||
blacksmith testbox download --id "$TBX" \
|
||||
"testbox-benchmark/$stage.$suffix" "$OUT/raw/$stage.$suffix" \
|
||||
>>"$OUT/$stage.download.log" 2>&1; then
|
||||
download_status=1
|
||||
fi
|
||||
done
|
||||
cat "$OUT/$stage.download.log"
|
||||
if (( run_status != 0 )); then
|
||||
return "$run_status"
|
||||
fi
|
||||
return "$download_status"
|
||||
}
|
||||
```
|
||||
|
||||
The helper supports exactly `first-clean`, `incremental-noop`, and
|
||||
`changed-file`. Each remote Cargo build is bounded to 20 minutes with a
|
||||
30-second kill grace period. It records a schema-2 JSON object for each stage
|
||||
containing:
|
||||
|
||||
* expected and observed source commit/tree identity before and after the build;
|
||||
* expected and observed Ghostty gitlink and initialized submodule HEAD;
|
||||
* clean/dirty file lists and source restoration status;
|
||||
* Testbox ID and adopted workflow run ID;
|
||||
* runner label, hostname, architecture, CPU count, and `uname`; the setup
|
||||
workflow fails closed unless the actual runner is x64 with 32 CPUs;
|
||||
* active Rust toolchain, `rustc`, Cargo, Zig, lockfile/toolchain hashes, and
|
||||
Ghostty package-manifest hash; and
|
||||
* Cargo exit status, `/usr/bin/time -p` values, and CLI transcript timing.
|
||||
|
||||
`first-clean` removes only the remote `cmux-tui/target` directory before a
|
||||
`cargo build -p cmux-tui --locked`. `incremental-noop` repeats that command
|
||||
without changing source. `changed-file` appends a comment to
|
||||
`cmux-tui/crates/cmux-tui/src/main.rs`, builds, and restores the original bytes
|
||||
before emitting its final record. A dirty or mismatched source before any
|
||||
stage, after restoration, or in the Ghostty submodule aborts the stage.
|
||||
|
||||
After all successful downloads, aggregate and verify the records:
|
||||
|
||||
```bash
|
||||
python3 - "$OUT" "$SOURCE_SHA" "$GHOSTTY_SHA" "$TBX" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
out = pathlib.Path(sys.argv[1])
|
||||
expected_source, expected_ghostty, testbox_id = sys.argv[2:]
|
||||
expected_tree = subprocess.check_output(
|
||||
["git", "rev-parse", f"{expected_source}^{{tree}}"], text=True
|
||||
).strip()
|
||||
required = {"first-clean", "incremental-noop", "changed-file"}
|
||||
records = []
|
||||
for path in sorted((out / "raw").glob("*.json")):
|
||||
record = json.loads(path.read_text(encoding="utf-8"))
|
||||
records.append(record)
|
||||
if {record.get("stage") for record in records} != required:
|
||||
raise SystemExit("timing evidence is missing one or more benchmark stages")
|
||||
for record in records:
|
||||
if record.get("testbox", {}).get("id") != testbox_id:
|
||||
raise SystemExit(f"{record.get('stage')} has the wrong Testbox ID")
|
||||
source = record.get("source", {})
|
||||
if source.get("expected_commit_sha") != expected_source or source.get("expected_tree_sha") != expected_tree:
|
||||
raise SystemExit(f"{record.get('stage')} has the wrong expected source identity")
|
||||
for side in ("before", "after"):
|
||||
snapshot = source.get(side, {})
|
||||
if snapshot.get("commit_sha") != expected_source or snapshot.get("tree_sha") != expected_tree:
|
||||
raise SystemExit(f"{record.get('stage')} has the wrong {side} source SHA")
|
||||
if snapshot.get("dirty_files"):
|
||||
raise SystemExit(f"{record.get('stage')} has dirty top-level source")
|
||||
ghostty = snapshot.get("ghostty", {})
|
||||
if ghostty.get("gitlink_sha") != expected_ghostty or ghostty.get("head_sha") != expected_ghostty:
|
||||
raise SystemExit(f"{record.get('stage')} has mismatched Ghostty identity")
|
||||
if ghostty.get("dirty_files"):
|
||||
raise SystemExit(f"{record.get('stage')} has dirty Ghostty source")
|
||||
if not record.get("ok"):
|
||||
raise SystemExit(f"{record.get('stage')} did not complete successfully")
|
||||
with (out / "timings.json").open("w", encoding="utf-8") as handle:
|
||||
json.dump({"schema": 2, "source_sha": expected_source, "ghostty_gitlink_sha": expected_ghostty, "testbox_id": testbox_id, "stages": records}, handle, indent=2, sort_keys=True)
|
||||
handle.write("\n")
|
||||
PY
|
||||
```
|
||||
|
||||
Keep `raw/*.json`, `raw/*.time`, `raw/*.log`, every `*.run.log` and download
|
||||
log, the setup artifact, and the source manifest in a new, unique
|
||||
`.cmux-scratch/` evidence directory. Never reuse a prior SHA-only directory;
|
||||
refuse to overwrite historical records. Do not add credentials or private keys.
|
||||
|
||||
## Fail-safe cleanup
|
||||
|
||||
Always download before cleanup. Use the checked-in cleanup helper rather than
|
||||
ignoring errors with `|| true`. After an independent operator decides the exact
|
||||
box may be destroyed, pass the ownership token generated with the warmup receipt
|
||||
and the literal `STOP`; never print the token:
|
||||
|
||||
```bash
|
||||
CLEANUP_TOKEN="${CLEANUP_TOKEN:-}"
|
||||
[[ "$CLEANUP_TOKEN" =~ ^[0-9a-f]{32}$ ]] || {
|
||||
echo "use the ownership token emitted by the warmup receipt" >&2
|
||||
exit 64
|
||||
}
|
||||
scripts/blacksmith-testbox-cleanup.sh "$TBX" "$OUT" "$CLEANUP_TOKEN" PREVIEW
|
||||
# Review cleanup-preview.json, then rerun with STOP:<sha256(cleanup-preview.json)>.
|
||||
```
|
||||
|
||||
It records a pre-stop status preview, stop result, post-stop status, and
|
||||
`list --all` output. The preview must match the receipt's workflow, job, and
|
||||
branch before any stop is attempted. Cleanup is destructive and requires a fresh `STOP:<sha256(cleanup-preview.json)>`
|
||||
confirmation after reviewing the current receipt-bound preview.
|
||||
It verifies that the specific Testbox ID is terminal or absent from the active
|
||||
inventory, accepts the known terminal states `completed`, `stopped`, `cancelled`,
|
||||
`failed`, `terminated`, and `hydration_failed`, plus a 409 saying the box is
|
||||
already stopped or completed, and polls for up to two minutes while cancellation
|
||||
propagates. Other stop, status, or list failures remain failures.
|
||||
Put it in an `EXIT` trap only after an independent operator exports
|
||||
`CONFIRM_TESTBOX_STOP_SHA` containing the SHA-256 of a separately reviewed
|
||||
`cleanup-preview.json`; otherwise preserve the benchmark's original exit status
|
||||
and leave the box for manual cleanup. The detailed benchmark writes a
|
||||
receipt and ownership token for the exact ID returned by warmup; cleanup refuses an ID
|
||||
or token that is not bound to that receipt. If warmup fails before returning an
|
||||
ID, retain before/after inventory but do not automatically stop a box, because
|
||||
an inventory diff cannot prove ownership across concurrent operators. Reconcile
|
||||
that orphan manually through the Blacksmith control plane.
|
||||
|
||||
## Timing interpretation
|
||||
|
||||
The benchmark reports two clocks. The local CLI transcript measures sync,
|
||||
transport, queueing, and the remote command. The downloaded `/usr/bin/time -p`
|
||||
record measures the remote `cargo build -p cmux-tui --locked` command. Compare
|
||||
remote `real` or `time_real_seconds` values for build performance, and retain
|
||||
CLI wall time when evaluating Testbox overhead.
|
||||
|
||||
`first-clean` is target-clean but dependency-warm: warmup runs `cargo fetch` and
|
||||
`zig build --fetch`, and the workflow may restore registry, git, and Zig caches.
|
||||
`incremental-noop` measures a second build on the same VM. `changed-file` is a
|
||||
controlled source change on that same VM. These are deliberately different
|
||||
from a cold-VM benchmark.
|
||||
|
||||
The existing 32-vCPU evidence at
|
||||
`.cmux-scratch/blacksmith-testbox-e40704611ac35f4ffa153/` remains historical
|
||||
provenance for setup SHA `e40704611ac35f0e3a806841a9eae383f4ffa153`, Testbox
|
||||
`tbx_01kzxebn91nhatkv4ygevh06vs`, and workflow run `31696013711`. Its raw
|
||||
records and cleanup result must not be rewritten when validating this hardening
|
||||
change.
|
||||
@@ -0,0 +1,473 @@
|
||||
# cmux-tui Testbox timing plan
|
||||
|
||||
Run this plan from the root of the isolated cmux worktree. It never invokes a
|
||||
Rust tool on the local Mac. Every `cargo`, `rustc`, and `zig` command below is
|
||||
inside a quoted command passed to `blacksmith testbox run`, or inside the
|
||||
setup-only GitHub job on the remote Linux runner.
|
||||
|
||||
## Fixed lane contract
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| Workflow | `.github/workflows/cmux-tui-testbox-warmup.yml` |
|
||||
| Job | `cmux-tui-rust` |
|
||||
| Runner | `blacksmith-32vcpu-ubuntu-2404` |
|
||||
| Protected environment | `blacksmith-testbox-trusted` |
|
||||
| Rust source of truth | `cmux-tui/rust-toolchain.toml`, via `./.github/actions/setup-cmux-tui-rust` |
|
||||
| Remote build helper | `scripts/blacksmith-cmux-tui-testbox-stage.sh` |
|
||||
| Cleanup helper | `scripts/blacksmith-testbox-cleanup.sh` |
|
||||
| Remote output | `testbox-benchmark/` |
|
||||
|
||||
A repository administrator must configure the protected environment with
|
||||
required reviewers, no secrets, administrator bypass disabled, and a deployment
|
||||
branch rule of exactly `main` before this plan is usable. The lane needs no
|
||||
environment variables: the workflow refuses any ref except `refs/heads/main`,
|
||||
so `main` alone decides what code runs in the token-bearing job. Verify that
|
||||
configuration before each run, and stop instead of treating the environment
|
||||
name as a guard if it drifts.
|
||||
`begin-testbox` exposes its auth token to commands in the Testbox, so
|
||||
`contents: read` is not a trust boundary and the token is not sandboxed. The
|
||||
repository does not currently pin a checksum-verified Blacksmith CLI artifact;
|
||||
that is a trusted-lane operational limitation. Use only the organization-
|
||||
approved CLI, record `blacksmith --version`, and stop rather than silently
|
||||
substituting a version.
|
||||
|
||||
The warmup job only checks out `main`, initializes `ghostty`,
|
||||
installs Linux headers/tools, installs the pinned Zig and Rust toolchains,
|
||||
fetches Cargo and Zig dependencies, and records JSON identity. `zig build
|
||||
--fetch` is the only build-system operation in warmup, and it exits before
|
||||
compiling. Rust builds happen only in the three explicit benchmark runs.
|
||||
|
||||
The current Blacksmith catalog reports the requested x64 label as 32 vCPU and
|
||||
121.6 GB, while the ARM label with the same vCPU count reports 96 GB. Keep the
|
||||
requested `blacksmith-32vcpu-ubuntu-2404` label unless repository Linux
|
||||
constraints make x64 impossible, and record the catalog result with the run.
|
||||
|
||||
## Exact benchmarked source and Ghostty identity
|
||||
|
||||
Warmup hydrates `main`. This section pins the separate commit you benchmark,
|
||||
which `blacksmith testbox run` synchronizes onto the warm box. A raw commit SHA
|
||||
is not a supported warmup ref (HTTP 422, `No ref found`), and this lane accepts
|
||||
only `main` regardless. Carry the benchmarked SHA as an assertion:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
git submodule update --init ghostty
|
||||
cd "$(git rev-parse --show-toplevel)"
|
||||
SOURCE_REF="$(git symbolic-ref --short HEAD)"
|
||||
if [[ ! "$SOURCE_REF" =~ ^[A-Za-z0-9._/-]+$ || "$SOURCE_REF" == *..* || "$SOURCE_REF" == */ || "$SOURCE_REF" == *//* ]]; then
|
||||
echo "HEAD must name a supported pushed branch ref" >&2
|
||||
exit 1
|
||||
fi
|
||||
SOURCE_SHA="$(git rev-parse HEAD)"
|
||||
SOURCE_TREE_SHA="$(git rev-parse 'HEAD^{tree}')"
|
||||
ghostty_entry="$(git ls-tree HEAD ghostty)"
|
||||
[[ "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ]] || {
|
||||
echo "HEAD:ghostty is not a gitlink" >&2
|
||||
exit 1
|
||||
}
|
||||
GHOSTTY_SHA="$(git rev-parse HEAD:ghostty)"
|
||||
[[ -n "$SOURCE_REF" ]]
|
||||
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
|
||||
[[ "$GHOSTTY_SHA" =~ ^[0-9a-f]{40}$ ]]
|
||||
[[ "$(git -C ghostty rev-parse HEAD)" == "$GHOSTTY_SHA" ]]
|
||||
[[ -z "$(git status --porcelain=v1 --untracked-files=normal)" ]]
|
||||
[[ -z "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]]
|
||||
remote_sha="$(git ls-remote --exit-code origin "refs/heads/$SOURCE_REF" | awk 'NR == 1 { print $1 }')"
|
||||
[[ "$remote_sha" == "$SOURCE_SHA" ]] || {
|
||||
echo "push the exact clean branch head before warming Testbox" >&2
|
||||
exit 1
|
||||
}
|
||||
EVIDENCE_ROOT="$PWD/.cmux-scratch"
|
||||
if [[ -e "$EVIDENCE_ROOT/blacksmith-testbox-$SOURCE_SHA" ]]; then
|
||||
RUN_SUFFIX="$(date -u +%Y%m%dT%H%M%SZ)-$$"
|
||||
else
|
||||
RUN_SUFFIX="initial-$$"
|
||||
fi
|
||||
OUT_ROOT="$EVIDENCE_ROOT/blacksmith-testbox-$SOURCE_SHA-$RUN_SUFFIX"
|
||||
if [[ -e "$OUT_ROOT" ]]; then
|
||||
echo "evidence directory already exists; choose a new run path: $OUT_ROOT" >&2
|
||||
exit 1
|
||||
fi
|
||||
mkdir -p "$OUT_ROOT/raw"
|
||||
python3 - "$SOURCE_REF" "$SOURCE_SHA" "$SOURCE_TREE_SHA" "$GHOSTTY_SHA" > "$OUT_ROOT/source.json" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
ref, sha, tree, ghostty = sys.argv[1:]
|
||||
print(json.dumps({
|
||||
"source_ref": ref,
|
||||
"source_sha": sha,
|
||||
"source_tree_sha": tree,
|
||||
"ghostty_gitlink_sha": ghostty,
|
||||
}, indent=2, sort_keys=True))
|
||||
PY
|
||||
|
||||
assert_source_unchanged() {
|
||||
local current_ref current_sha current_tree current_ghostty remote_sha ghostty_entry
|
||||
current_ref="$(git symbolic-ref --short HEAD)"
|
||||
current_sha="$(git rev-parse HEAD)"
|
||||
current_tree="$(git rev-parse 'HEAD^{tree}')"
|
||||
ghostty_entry="$(git ls-tree HEAD ghostty)"
|
||||
current_ghostty="$(git rev-parse HEAD:ghostty)"
|
||||
remote_sha="$(git ls-remote --exit-code origin "refs/heads/$SOURCE_REF" | awk 'NR == 1 { print $1 }')"
|
||||
if [[ "$current_ref" != "$SOURCE_REF" || "$current_sha" != "$SOURCE_SHA" ||
|
||||
"$current_tree" != "$SOURCE_TREE_SHA" || "$current_ghostty" != "$GHOSTTY_SHA" ||
|
||||
! "$ghostty_entry" =~ ^160000[[:space:]]commit[[:space:]][0-9a-f]{40}[[:space:]]ghostty$ ||
|
||||
"$remote_sha" != "$SOURCE_SHA" ||
|
||||
-n "$(git status --porcelain=v1 --untracked-files=normal)" ||
|
||||
-n "$(git -C ghostty status --porcelain=v1 --untracked-files=normal)" ]]; then
|
||||
echo "source branch, tree, Ghostty gitlink, remote head, or clean status changed" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
assert_source_unchanged
|
||||
```
|
||||
|
||||
`assert_source_unchanged` runs before every stage below. If the branch moved,
|
||||
the worktree became dirty, or the Ghostty pointer changed, stop the box and
|
||||
start a new evidence directory. Do not silently substitute the new SHA.
|
||||
|
||||
## Warmup and setup identity
|
||||
|
||||
Use the branch ref, not `SOURCE_SHA`, in warmup:
|
||||
|
||||
```bash
|
||||
WORKFLOW=.github/workflows/cmux-tui-testbox-warmup.yml
|
||||
JOB=cmux-tui-rust
|
||||
OUT_ROOT="${OUT_ROOT:?set by the exact-source preflight above}"
|
||||
OUT="$OUT_ROOT"
|
||||
TBX=""
|
||||
warmup_testbox_id=""
|
||||
cleanup_token=""
|
||||
before_list_status=125
|
||||
mkdir -p "$OUT/raw"
|
||||
cleanup() {
|
||||
local result=$?
|
||||
local cleanup_status=0
|
||||
local after_list_status=125
|
||||
trap - EXIT
|
||||
if [[ -n "$TBX" && -n "$cleanup_token" && -n "${CONFIRM_TESTBOX_STOP_SHA:-}" ]]; then
|
||||
set +e
|
||||
scripts/blacksmith-testbox-cleanup.sh "$TBX" "$OUT" "$cleanup_token" "STOP:${CONFIRM_TESTBOX_STOP_SHA}"
|
||||
cleanup_status=$?
|
||||
set -e
|
||||
else
|
||||
# Without the CLI receipt there is no proof that a newly listed box belongs
|
||||
# to this invocation. Report inventory, but never stop another operator's box.
|
||||
set +e
|
||||
./scripts/blacksmith-bounded-command.sh 60 \
|
||||
blacksmith testbox list --all >"$OUT/list-after-warmup-failure.log" 2>&1
|
||||
after_list_status=$?
|
||||
set -e
|
||||
if (( after_list_status != 0 )); then
|
||||
cleanup_status="$after_list_status"
|
||||
echo "could not capture post-failure Testbox inventory" >&2
|
||||
else
|
||||
cleanup_status=1
|
||||
echo "warmup returned no owned Testbox receipt; no automatic stop was attempted" >&2
|
||||
fi
|
||||
fi
|
||||
if (( result == 0 && cleanup_status != 0 )) && [[ -n "${CONFIRM_TESTBOX_STOP_SHA:-}" ]]; then
|
||||
result="$cleanup_status"
|
||||
fi
|
||||
exit "$result"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
blacksmith auth whoami
|
||||
blacksmith --version >"$OUT/blacksmith-version.txt"
|
||||
cat "$OUT/blacksmith-version.txt"
|
||||
blacksmith runners catalog >"$OUT/runner-catalog.json"
|
||||
set +e
|
||||
./scripts/blacksmith-bounded-command.sh 60 \
|
||||
blacksmith testbox list --all >"$OUT/list-before-warmup.log" 2>&1
|
||||
before_list_status=$?
|
||||
set -e
|
||||
cat "$OUT/list-before-warmup.log"
|
||||
if (( before_list_status != 0 )); then
|
||||
echo "refusing to warm a Testbox without a baseline inventory" >&2
|
||||
exit "$before_list_status"
|
||||
fi
|
||||
set +e
|
||||
./scripts/blacksmith-bounded-command.sh 1200 \
|
||||
blacksmith testbox warmup "$WORKFLOW" \
|
||||
--ref main \
|
||||
--job "$JOB" \
|
||||
--idle-timeout 30 \
|
||||
>"$OUT/warmup.log" 2>&1
|
||||
warmup_status=$?
|
||||
set -e
|
||||
cat "$OUT/warmup.log"
|
||||
if (( warmup_status != 0 )); then
|
||||
# Do not parse IDs from a failed CLI transcript. It may contain a stale ID
|
||||
# from an error message, and cleanup is intentionally receipt-bound.
|
||||
exit "$warmup_status"
|
||||
fi
|
||||
set +e
|
||||
warmup_testbox_id="$(python3 - "$OUT/warmup.log" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
|
||||
text = open(sys.argv[1], encoding="utf-8").read()
|
||||
ids = re.findall(r"\btbx_[A-Za-z0-9_-]+\b", text)
|
||||
if not ids:
|
||||
raise SystemExit("warmup output did not contain a Testbox ID")
|
||||
print(ids[-1])
|
||||
PY
|
||||
)"
|
||||
parse_status=$?
|
||||
set -e
|
||||
if (( parse_status != 0 )); then
|
||||
exit "$parse_status"
|
||||
fi
|
||||
umask 077
|
||||
set +e
|
||||
cleanup_token="$(python3 - "$OUT/testbox-receipt.json" "$warmup_testbox_id" "$WORKFLOW" "$JOB" "$SOURCE_REF" "$SOURCE_SHA" "$SOURCE_TREE_SHA" "$GHOSTTY_SHA" <<'PY'
|
||||
import datetime as dt
|
||||
import json
|
||||
import pathlib
|
||||
import secrets
|
||||
import sys
|
||||
|
||||
path, testbox_id, workflow, job, source_ref, source_sha, source_tree, ghostty_sha = sys.argv[1:]
|
||||
token = secrets.token_hex(16)
|
||||
path = pathlib.Path(path)
|
||||
path.write_text(json.dumps({
|
||||
"schema": 1,
|
||||
"testbox_id": testbox_id,
|
||||
"workflow": workflow,
|
||||
"job": job,
|
||||
"source_ref": source_ref,
|
||||
"source_sha": source_sha,
|
||||
"source_tree_sha": source_tree,
|
||||
"ghostty_gitlink_sha": ghostty_sha,
|
||||
"confirmation_token": token,
|
||||
"created_at": dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
}, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
path.chmod(0o600)
|
||||
print(token)
|
||||
PY
|
||||
)"
|
||||
receipt_status=$?
|
||||
set -e
|
||||
if (( receipt_status != 0 )); then
|
||||
echo "could not create the warmup ownership receipt" >&2
|
||||
exit "$receipt_status"
|
||||
fi
|
||||
TBX="$warmup_testbox_id"
|
||||
printf 'Testbox ID: %s\n' "$TBX" | tee "$OUT/testbox-id.txt"
|
||||
if (( warmup_status != 0 )); then
|
||||
exit "$warmup_status"
|
||||
fi
|
||||
set +e
|
||||
blacksmith testbox status --id "$TBX" --wait --wait-timeout 15m \
|
||||
>"$OUT/status-ready.log" 2>&1
|
||||
status_ready=$?
|
||||
set -e
|
||||
cat "$OUT/status-ready.log"
|
||||
if (( status_ready != 0 )); then
|
||||
exit "$status_ready"
|
||||
fi
|
||||
```
|
||||
|
||||
The workflow validates that the dispatch ref exactly matches the protected
|
||||
reviewed branch, that `github.sha` exactly matches the protected reviewed SHA,
|
||||
and that the remote branch still resolves to that SHA. If a direct GitHub
|
||||
dispatch supplies the optional `source_sha` input, it is an assertion and must
|
||||
equal the protected SHA. The Blacksmith CLI path needs no arbitrary workflow
|
||||
input: it supplies `testbox_id`, while `--ref` selects the explicitly reviewed
|
||||
branch and `github.sha` carries the pinned candidate.
|
||||
The workflow concurrency group serializes setup requests by Testbox ID, even
|
||||
when source SHAs differ. The remote `flock` begins after Blacksmith's rsync, so
|
||||
it protects stage/build/artifact writes only. Blacksmith exposes no pre-rsync
|
||||
lease; one Testbox ID must have one owning worktree/operator, and independent
|
||||
clients must not issue concurrent `run` or download commands. This is an
|
||||
explicit trusted-lane limitation.
|
||||
|
||||
Do not issue a separate interpolated identity command. The setup job's
|
||||
`setup-identity.json` artifact and each stage helper's pre-build JSON record are
|
||||
the identity transcripts. They describe different commits on purpose: the setup
|
||||
artifact names the hydrated `main`, and the stage record names the benchmarked
|
||||
revision you synchronized. The helper verifies the Testbox VM marker, claimed
|
||||
Testbox ID, the hydration marker's own internal consistency and runner class,
|
||||
and then the synchronized source commit/tree, Ghostty gitlink/checkout, and
|
||||
clean status before it invokes Cargo, repeating the source checks after the
|
||||
build. Each stage JSON carries a `hydration` block with the warmed ref and
|
||||
commit plus `matches_benchmarked_source`, which is normally `false`. Keep the
|
||||
setup artifact URL or download it into `$OUT`. A successful setup copies the
|
||||
same JSON to `/tmp/.testbox/cmux-tui-rust-setup-identity.json`; the stage helper
|
||||
rejects a missing or malformed marker, so failed hydration cannot be
|
||||
benchmarked. The active Rust, Cargo, and Zig versions must still equal the
|
||||
hydrated ones, so a branch that repins its toolchain stops the run instead of
|
||||
reporting a cold-cache timing.
|
||||
|
||||
## Three remote build timings
|
||||
|
||||
The helper creates one structured JSON record, one raw Cargo log, and one raw
|
||||
`/usr/bin/time -p` file per stage. Each remote Cargo build is bounded to 20
|
||||
minutes with a 30-second kill grace period, and the outer CLI invocation is
|
||||
bounded to 25 minutes so rsync, SSH, or control-plane hangs cannot bypass the
|
||||
benchmark's cleanup path. It verifies source and submodule identity before the
|
||||
stage, exports and records the exact Zig binary used by Cargo, holds a remote
|
||||
`flock` through all writes, restores the controlled changed file from an
|
||||
integrity-checked backup, and verifies clean identity again.
|
||||
|
||||
```bash
|
||||
# Run this orchestration block in Bash, not an interactive zsh session.
|
||||
run_stage() {
|
||||
local stage="$1"
|
||||
local run_status download_status=0
|
||||
set +e
|
||||
printf -v remote_command \
|
||||
'CMUX_TESTBOX_REMOTE=1 CMUX_TESTBOX_ID=%q %q %q %q %q' \
|
||||
"$TBX" ./scripts/blacksmith-cmux-tui-testbox-stage.sh \
|
||||
"$stage" "$SOURCE_SHA" "$GHOSTTY_SHA"
|
||||
./scripts/blacksmith-bounded-command.sh 1500 blacksmith testbox run --id "$TBX" --debug \
|
||||
"$remote_command" >"$OUT/$stage.run.log" 2>&1
|
||||
run_status=$?
|
||||
set -e
|
||||
cat "$OUT/$stage.run.log"
|
||||
|
||||
# rsync --delete can remove remote output before the next run. Download each
|
||||
# stage immediately, before starting another stage.
|
||||
: >"$OUT/$stage.download.log"
|
||||
for suffix in json time log; do
|
||||
if ! ./scripts/blacksmith-bounded-command.sh 120 \
|
||||
blacksmith testbox download --id "$TBX" \
|
||||
"testbox-benchmark/$stage.$suffix" "$OUT/raw/$stage.$suffix" \
|
||||
>>"$OUT/$stage.download.log" 2>&1; then
|
||||
download_status=1
|
||||
fi
|
||||
done
|
||||
cat "$OUT/$stage.download.log"
|
||||
if (( run_status != 0 )); then
|
||||
return "$run_status"
|
||||
fi
|
||||
return "$download_status"
|
||||
}
|
||||
|
||||
benchmark_status=0
|
||||
for stage in first-clean incremental-noop changed-file; do
|
||||
if ! assert_source_unchanged; then
|
||||
benchmark_status=1
|
||||
break
|
||||
fi
|
||||
if ! run_stage "$stage"; then
|
||||
benchmark_status=1
|
||||
break
|
||||
fi
|
||||
done
|
||||
if (( benchmark_status != 0 )); then
|
||||
exit "$benchmark_status"
|
||||
fi
|
||||
```
|
||||
|
||||
`first-clean` is target-clean but dependency-warm. `incremental-noop` repeats
|
||||
the exact build on the same VM. `changed-file` appends a comment to
|
||||
`cmux-tui/crates/cmux-tui/src/main.rs`, builds, and restores the original bytes.
|
||||
The local worktree is never mutated by the helper.
|
||||
|
||||
Verify the downloaded records before accepting timings:
|
||||
|
||||
```bash
|
||||
python3 - "$OUT" "$SOURCE_SHA" "$GHOSTTY_SHA" "$TBX" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
out = pathlib.Path(sys.argv[1])
|
||||
expected_source, expected_ghostty, testbox_id = sys.argv[2:]
|
||||
expected_tree = subprocess.check_output(
|
||||
["git", "rev-parse", f"{expected_source}^{{tree}}"], text=True
|
||||
).strip()
|
||||
required = {"first-clean", "incremental-noop", "changed-file"}
|
||||
records = [json.loads(path.read_text(encoding="utf-8")) for path in sorted((out / "raw").glob("*.json"))]
|
||||
if {record.get("stage") for record in records} != required:
|
||||
raise SystemExit("expected exactly three stage records")
|
||||
for record in records:
|
||||
stage = record.get("stage")
|
||||
runner = record.get("runner", {})
|
||||
if runner.get("arch") != "x86_64" or runner.get("cpu_count") != 32:
|
||||
raise SystemExit(f"{stage}: wrong runner identity {runner}")
|
||||
if record.get("testbox", {}).get("id") != testbox_id:
|
||||
raise SystemExit(f"{stage}: wrong Testbox ID")
|
||||
source_record = record.get("source", {})
|
||||
if source_record.get("expected_commit_sha") != expected_source or source_record.get("expected_tree_sha") != expected_tree:
|
||||
raise SystemExit(f"{stage}: wrong expected source identity")
|
||||
for side in ("before", "after"):
|
||||
source = source_record.get(side, {})
|
||||
if source.get("commit_sha") != expected_source or source.get("tree_sha") != expected_tree or source.get("dirty_files"):
|
||||
raise SystemExit(f"{stage}: source mismatch or dirty {side} checkout")
|
||||
ghostty = source.get("ghostty", {})
|
||||
if ghostty.get("gitlink_sha") != expected_ghostty or ghostty.get("head_sha") != expected_ghostty or ghostty.get("dirty_files"):
|
||||
raise SystemExit(f"{stage}: Ghostty mismatch or dirty {side} checkout")
|
||||
if not record.get("ok"):
|
||||
raise SystemExit(f"{stage}: build failed")
|
||||
with (out / "timings.json").open("w", encoding="utf-8") as handle:
|
||||
json.dump({"schema": 2, "source_sha": expected_source, "ghostty_gitlink_sha": expected_ghostty, "testbox_id": testbox_id, "stages": records}, handle, indent=2, sort_keys=True)
|
||||
handle.write("\n")
|
||||
PY
|
||||
```
|
||||
|
||||
## Cleanup and evidence
|
||||
|
||||
Download all raw files and `timings.json` before cleanup. Then, after an
|
||||
operator explicitly decides this exact box may be destroyed, call the fail-safe
|
||||
helper. It previews the exact receipt context, preserves an already-completed
|
||||
409, polls cancellation to a bounded deadline, fails on other stop/status/list
|
||||
errors, and verifies this exact Testbox ID is no longer active:
|
||||
|
||||
```bash
|
||||
cleanup_token="${cleanup_token:-}"
|
||||
[[ "$cleanup_token" =~ ^[0-9a-f]{32}$ ]] || {
|
||||
echo "use the confirmation token emitted by the warmup receipt" >&2
|
||||
exit 64
|
||||
}
|
||||
scripts/blacksmith-testbox-cleanup.sh "$TBX" "$OUT" "$cleanup_token" PREVIEW
|
||||
# Review cleanup-preview.json, then rerun with STOP:<sha256(cleanup-preview.json)>.
|
||||
```
|
||||
|
||||
A shell `EXIT` trap may call that helper only when an independent operator has
|
||||
exported `CONFIRM_TESTBOX_STOP_SHA` with the SHA-256 of a separately reviewed
|
||||
`cleanup-preview.json`; otherwise it preserves the benchmark status, records
|
||||
inventory, and leaves the box for explicit manual cleanup.
|
||||
Warmup writes `testbox-receipt.json` and an ownership token bound to the exact
|
||||
returned ID; cleanup refuses a mismatched ID or token. If warmup fails before
|
||||
returning an ID, retain before/after inventories but do not automatically stop a
|
||||
box, because an inventory diff cannot prove ownership across concurrent
|
||||
operators. Reconcile that orphan manually through the
|
||||
Blacksmith control plane. Keep both inventories and their command statuses,
|
||||
plus warmup/status/identity transcripts, every stage run and download
|
||||
transcript, raw JSON/time/log files, runner catalog, setup identity artifact,
|
||||
cleanup logs, the receipt, and the final source manifest in the new, unique
|
||||
`.cmux-scratch/` directory. Never reuse a prior SHA-only directory or overwrite
|
||||
historical records. Never store credentials, private keys, or
|
||||
`/tmp/.testbox/auth_token`.
|
||||
|
||||
Record these fields alongside `timings.json`:
|
||||
|
||||
1. Exact source branch, full source SHA/tree SHA, Ghostty gitlink SHA, and the
|
||||
clean-status result before each stage.
|
||||
2. Requested runner label and catalog output. The setup job rejects any
|
||||
actual architecture or CPU count other than x64 and 32.
|
||||
3. Blacksmith CLI version, Testbox ID, setup workflow run/job IDs, identity run
|
||||
ID, and each stage run/sync ID from raw transcripts.
|
||||
4. Whether the comparison was target-clean, registry/git-cache warm,
|
||||
Zig-cache warm, or a genuinely cold VM. Warmup deliberately hydrates
|
||||
dependencies, so `first-clean` is target-cold and dependency-warm.
|
||||
5. Cleanup stop/status/list output and whether the specific ID was absent from
|
||||
the active inventory.
|
||||
|
||||
The historical 32-vCPU evidence at
|
||||
`.cmux-scratch/blacksmith-testbox-e40704611ac35f4ffa153/` remains unchanged and
|
||||
must never be selected as a writable `OUT` directory:
|
||||
setup SHA `e40704611ac35f0e3a806841a9eae383f4ffa153`, Testbox
|
||||
`tbx_01kzxebn91nhatkv4ygevh06vs`, workflow run `31696013711`, first-clean
|
||||
`161.47s`, incremental no-op `8.28s`, changed-file `9.13s`, and cleanup with no
|
||||
active box. Do not rewrite it while validating this hardening change.
|
||||
|
||||
Prior hosted cmux-tui correctness runs without Cargo durations are provenance,
|
||||
not performance comparisons. Prior Blacksmith macOS Swift/Xcode artifacts use
|
||||
a different OS, architecture, runner SKU, cache state, and workload, so they
|
||||
are context rather than a Rust baseline.
|
||||
Reference in New Issue
Block a user