* test: red regression for termination watchdog (#6758) cmux can hang the main thread for ~30s on Cmd+Q when a clipboard-history manager (Paste, Raycast, Maccy, …) is mid-read of cmux's promised pasteboard data: AppKit's will-terminate gauntlet runs CFPasteboardResolveAllPromisedData, which blocks on a stuck mach round-trip to the pasteboard server until the OS force-kills the app. This is the third "an observer blocks the main thread during quit" report (cf. #6415 PostHog flush, #6381 ghostty lock); the structural gap is that quit has no global "return within N seconds no matter what" guard. Add TerminationWatchdog plus its tests, with the watchdog deliberately inert (it never starts the firing thread) so the tests go red. The end-to- end pasteboard deadlock is not unit-testable — reproducing it requires the real pasteboard server and would wedge the test process — so the tests cover the watchdog mechanism that bounds it. The fix commit starts the thread and arms the watchdog from the terminate path. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Bound app termination with a force-exit watchdog (#6758) Implement TerminationWatchdog.arm and arm it from the terminate path so a committed quit always returns within a bounded time, even when AppKit's will-terminate gauntlet wedges on an Apple-owned observer we don't control (CFPasteboardResolveAllPromisedData blocking on a stuck pasteboard-server round-trip while a clipboard-history manager reads cmux's promised data). The watchdog runs on a dedicated background thread with no run-loop, GCD, or main-actor dependency, so it fires even while the main thread is parked in mach_msg. It is armed in prepareForConfirmedAppTermination() — after the critical session/state save and before AppKit posts will-terminate — and, as a backstop, at the start of applicationWillTerminate(). Arming is idempotent, so the two sites and repeated quit attempts never stack threads. If the process has not exited within the deadline it force-exits cleanly, turning a ~30s hang into a bounded quit. This closes the structural gap shared with #6415 and #6381: quit now has a global "return within N seconds no matter what" guard. Fixes #6758 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Address review: lock-free watchdog exit, drop singleton (#6758) - Codex/autoreview P1 (correctness): the watchdog's onFire logged a StartupBreadcrumbLog entry (flock + Foundation/file I/O) before _exit. If that logging stalled or contended during an already-wedged termination, the watchdog thread could block before reaching _exit and the quit hang would stay unbounded — defeating the guarantee. Drop the breadcrumb: the firing path is now an unconditional, lock-free _exit (the default onFire), which does zero Foundation/filesystem work before exiting. - Greptile P1 (no-ambient-global-state): replace the TerminationWatchdog.shared singleton with an AppDelegate-owned instance, next to the existing terminate-control state (terminateKillWatchdogTask). The type was already injectable, so this is a small wiring change. - Greptile P2: document why the deadline uses a raw Thread + Thread.sleep rather than a GCD timer (the wedged termination can sit on GCD/run-loop infrastructure, so the firing path must not depend on it). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Document lock-not-actor choice in TerminationWatchdog (#6758) cmux-policy (Aziz concurrency) prefers actor isolation over locks for new runtime state. Rejected here with rationale recorded in-code: an actor would force `arm()` async, but it is called synchronously from the terminate delegate methods and the deadline fires on a raw Thread — and the watchdog must not depend on the Swift concurrency runtime, which may itself be wedged during the termination it guards against. This is the same sanctioned NSLock + nonisolated(unsafe) shape TerminalPasteboardService uses for synchronous- callback state. Comment-only change. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Determinize TerminationWatchdog test via injected scheduler (#6758) CI's test-determinism gate (scripts/check-test-determinism.py --strict) flagged the prior tests for real sleeps / wall-clock timeouts (sleep-then-assert and assert-on-duration). Invert the time dependency per the gate's contract instead of allowlisting: extract the deadline scheduler as an injectable `DeadlineScheduler`. Production keeps the raw background Thread (`TerminationWatchdog.threadScheduler`); the tests inject a synchronous capturing scheduler and advance the deadline by hand. The tests now assert idempotency (three arms schedule the deadline exactly once) and exactly-once firing with zero real sleeps, timeouts, or wall-clock reads, so they are deterministic by construction. arm() is now a thin idempotent latch over scheduleDeadline(deadline, onFire); behavior is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * Use atomic termination watchdog latch * Use non-deprecated termination watchdog latch * Use C11 atomic termination watchdog latch * Save termination state before watchdog fallback * Avoid growing AppDelegate termination path --------- Co-authored-by: cmux <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Austin Wang
co-authored by
Claude Opus 4.8
cmux
Claude Opus 4.8
cmux
parent
5265559d59
commit
39df334577
@@ -2,7 +2,7 @@
|
||||
# Format: max_lines<TAB>relative path
|
||||
# Reduce counts as files shrink. CI fails if tracked files exceed this budget.
|
||||
34658 CLI/cmux.swift
|
||||
17892 Sources/AppDelegate.swift
|
||||
17907 Sources/AppDelegate.swift
|
||||
16491 Sources/ContentView.swift
|
||||
14239 Sources/TerminalController.swift
|
||||
12921 Sources/Workspace.swift
|
||||
|
||||
|
@@ -1101,6 +1101,9 @@ final class AppDelegate: NSObject, NSApplicationDelegate, UNUserNotificationCent
|
||||
// True while remote tmux kill-before-quit owns the terminate reply.
|
||||
private var isAwaitingTerminateKills = false
|
||||
private var terminateKillWatchdogTask: Task<Void, Never>?
|
||||
/// Hard deadline that force-exits if AppKit's terminate gauntlet wedges on an
|
||||
/// observer we don't own (e.g. CFPasteboardResolveAllPromisedData, #6758).
|
||||
private let terminationWatchdog = TerminationWatchdog()
|
||||
private var activeQuitConfirmationAlertPresenter: QuitConfirmationAlertPresenter?
|
||||
private var activeQuitConfirmationOwnsTerminateRequest = false
|
||||
private var didInstallLifecycleSnapshotObservers = false
|
||||
@@ -1871,6 +1874,12 @@ final class AppDelegate: NSObject, NSApplicationDelegate, UNUserNotificationCent
|
||||
isTerminatingApp = true
|
||||
_ = saveSessionSnapshotIncludingProcessDetectedIndexes(includeScrollback: true, removeWhenEmpty: false)
|
||||
ClosedItemHistoryStore.shared.flushPendingSaves()
|
||||
// Quit is committed and the critical state is now on disk. Bound the
|
||||
// remainder of the terminate sequence so a blocked Apple will-terminate
|
||||
// observer (e.g. CFPasteboardResolveAllPromisedData, #6758) can't hang
|
||||
// the main thread for ~30s. Idempotent and a no-op if the process exits
|
||||
// first.
|
||||
terminationWatchdog.arm()
|
||||
}
|
||||
|
||||
private func presentQuitConfirmationAlert(
|
||||
@@ -1984,16 +1993,22 @@ final class AppDelegate: NSObject, NSApplicationDelegate, UNUserNotificationCent
|
||||
|
||||
func applicationWillTerminate(_ notification: Notification) {
|
||||
StartupBreadcrumbLog.append("appDelegate.willTerminate.begin")
|
||||
sentryStopMemoryContextRefresh()
|
||||
// Backstop for any terminate path that did not route through
|
||||
// prepareForConfirmedAppTermination() (idempotent with the primary arm).
|
||||
// Apple's promised-pasteboard observer can fire before this delegate
|
||||
// method, so the primary arm above is what bounds #6758; this only
|
||||
// widens coverage to other entrypoints.
|
||||
isTerminatingApp = true
|
||||
_ = saveSessionSnapshotIncludingProcessDetectedIndexes(includeScrollback: true, removeWhenEmpty: false)
|
||||
ClosedItemHistoryStore.shared.flushPendingSaves()
|
||||
terminationWatchdog.arm()
|
||||
sentryStopMemoryContextRefresh()
|
||||
// Plain quit detaches local ssh clients; explicit close already killed marked sessions.
|
||||
remoteTmuxController.detachAll()
|
||||
// Best-effort presence goodbye; unclean exits are covered by the
|
||||
// service's missed-heartbeat timeout.
|
||||
PresenceHeartbeatClient.shared.appWillTerminate()
|
||||
closeAllWebInspectorsBeforeAppTeardown()
|
||||
_ = saveSessionSnapshotIncludingProcessDetectedIndexes(includeScrollback: true, removeWhenEmpty: false)
|
||||
ClosedItemHistoryStore.shared.flushPendingSaves()
|
||||
stopSessionAutosaveTimer()
|
||||
CloudVMActionLauncher.shared.terminateAll()
|
||||
CmuxSSHURLProcessLauncher.shared.terminateAll()
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
import Darwin
|
||||
import Foundation
|
||||
|
||||
/// Hard deadline on AppKit's application-termination sequence.
|
||||
///
|
||||
/// `-[NSApplication terminate:]` synchronously posts `NSApplicationWillTerminate`
|
||||
/// and drives a gauntlet of observers — several of them Apple's own and outside
|
||||
/// our control. One is `CFPasteboardResolveAllPromisedData`, which flushes
|
||||
/// promised (lazy) pasteboard data with a blocking mach round-trip to the
|
||||
/// pasteboard server. When a clipboard-history manager (Paste, Raycast, Maccy,
|
||||
/// Pastebot, …) is mid-read of cmux's promised clipboard data, that round-trip
|
||||
/// can wedge for ~30s on the main thread until the OS force-kills the app
|
||||
/// (https://github.com/manaflow-ai/cmux/issues/6758). The same structural gap —
|
||||
/// quit having no global "return within N seconds no matter what" guard —
|
||||
/// produced #6415 (`PostHogAnalytics.flush()`) and #6381 (`ghostty` lock).
|
||||
///
|
||||
/// This watchdog closes that gap. It runs on a dedicated background thread with
|
||||
/// no run-loop, GCD-queue, or main-actor dependency, so it fires even while the
|
||||
/// main thread is parked in `mach_msg`. Arm it the instant the app commits to
|
||||
/// quitting; if the process has not exited within `deadline`, it force-exits,
|
||||
/// turning a multi-second hang into a bounded quit. The firing path is
|
||||
/// deliberately unconditional and lock-free — it does no Foundation/filesystem
|
||||
/// work before exiting, because the termination it guards against may itself be
|
||||
/// wedged on exactly such a lock. cmux's critical session/state save runs
|
||||
/// synchronously *before* the watchdog is armed, so the bytes that matter are
|
||||
/// already on disk if the deadline ever fires.
|
||||
///
|
||||
/// Not a singleton: the app's lifecycle owner (`AppDelegate`) holds the
|
||||
/// instance, alongside its other terminate-control state.
|
||||
final class TerminationWatchdog: Sendable {
|
||||
/// Budget for the committed-quit sequence (remote-session kill defer plus
|
||||
/// AppKit's will-terminate gauntlet). Normal teardown finishes in well under
|
||||
/// a second; this leaves generous headroom while still beating the OS's
|
||||
/// ~30s hang watchdog by a wide margin.
|
||||
static let defaultDeadline: TimeInterval = 8
|
||||
|
||||
/// Schedules `fire` to run once after `deadline` seconds. Injectable so tests
|
||||
/// advance the deadline by hand instead of sleeping for real.
|
||||
typealias DeadlineScheduler =
|
||||
@Sendable (_ deadline: TimeInterval, _ fire: @escaping @Sendable () -> Void) -> Void
|
||||
|
||||
/// Production scheduler: a raw `Thread` that sleeps then fires. A raw Thread,
|
||||
/// deliberately NOT a GCD queue or `DispatchSourceTimer` — the wedged
|
||||
/// termination this guards against can sit on Foundation, GCD, or run-loop
|
||||
/// infrastructure, so the firing path must depend on none of it. The thread
|
||||
/// parks only during the brief quit window and is reclaimed when the process
|
||||
/// exits (the common path, well before the deadline).
|
||||
static let threadScheduler: DeadlineScheduler = { deadline, fire in
|
||||
let thread = Thread {
|
||||
Thread.sleep(forTimeInterval: deadline)
|
||||
fire()
|
||||
}
|
||||
thread.name = "com.cmuxterm.termination-watchdog"
|
||||
thread.stackSize = 128 * 1024
|
||||
thread.start()
|
||||
}
|
||||
|
||||
// C11 atomic, not an actor: `arm()` is called synchronously from terminate
|
||||
// delegate methods and must not depend on Swift concurrency while guarding a
|
||||
// wedged termination path. This is a one-shot 0 -> 1 latch; the deadline
|
||||
// callback itself remains lock-free.
|
||||
nonisolated(unsafe) private var latch = CMUXTerminationWatchdogLatchMake()
|
||||
private let onFire: @Sendable () -> Void
|
||||
private let scheduleDeadline: DeadlineScheduler
|
||||
|
||||
/// - Parameters:
|
||||
/// - onFire: invoked at most once, on the scheduler's thread, when the
|
||||
/// deadline elapses. It MUST stay lock-free and non-blocking — it runs
|
||||
/// precisely when termination is suspected to be wedged, so any
|
||||
/// Foundation, filesystem, or lock work before exiting could itself stall
|
||||
/// and reintroduce the unbounded hang. The default is a bare `_exit`.
|
||||
/// - scheduleDeadline: how the deadline is scheduled. Defaults to
|
||||
/// ``threadScheduler``; tests inject a synchronous capture so they can
|
||||
/// advance the deadline by hand.
|
||||
init(
|
||||
onFire: @escaping @Sendable () -> Void = { _exit(EXIT_SUCCESS) },
|
||||
scheduleDeadline: @escaping DeadlineScheduler = TerminationWatchdog.threadScheduler
|
||||
) {
|
||||
self.onFire = onFire
|
||||
self.scheduleDeadline = scheduleDeadline
|
||||
}
|
||||
|
||||
/// Arms the one-shot deadline. Idempotent: repeated calls — multiple quit
|
||||
/// attempts, or several commit sites arming for one request — schedule the
|
||||
/// deadline only once, so `onFire` runs at most once.
|
||||
func arm(deadline: TimeInterval = TerminationWatchdog.defaultDeadline) {
|
||||
guard CMUXTerminationWatchdogLatchClaim(&latch) else {
|
||||
return
|
||||
}
|
||||
|
||||
scheduleDeadline(deadline, onFire)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
#include "TerminationWatchdogAtomic.h"
|
||||
|
||||
CMUXTerminationWatchdogLatch CMUXTerminationWatchdogLatchMake(void) {
|
||||
CMUXTerminationWatchdogLatch latch;
|
||||
atomic_init(&latch.isArmed, false);
|
||||
return latch;
|
||||
}
|
||||
|
||||
bool CMUXTerminationWatchdogLatchClaim(CMUXTerminationWatchdogLatch *latch) {
|
||||
bool expected = false;
|
||||
return atomic_compare_exchange_strong_explicit(
|
||||
&latch->isArmed,
|
||||
&expected,
|
||||
true,
|
||||
memory_order_acq_rel,
|
||||
memory_order_acquire
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
#ifndef CMUX_TERMINATION_WATCHDOG_ATOMIC_H
|
||||
#define CMUX_TERMINATION_WATCHDOG_ATOMIC_H
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdatomic.h>
|
||||
|
||||
typedef struct {
|
||||
atomic_bool isArmed;
|
||||
} CMUXTerminationWatchdogLatch;
|
||||
|
||||
CMUXTerminationWatchdogLatch CMUXTerminationWatchdogLatchMake(void);
|
||||
bool CMUXTerminationWatchdogLatchClaim(CMUXTerminationWatchdogLatch *latch);
|
||||
|
||||
#endif
|
||||
@@ -1 +1,2 @@
|
||||
@import GhosttyKit;
|
||||
#import "Sources/TerminationWatchdogAtomic.h"
|
||||
|
||||
@@ -1058,6 +1058,9 @@
|
||||
5154BEAB50364B86A9E36E4B /* TerminalViewportUITestRecorder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 31B04B98376C9BA8B9E44DCB /* TerminalViewportUITestRecorder.swift */; };
|
||||
A5001532 /* TerminalWindowPortal.swift in Sources */ = {isa = PBXBuildFile; fileRef = A5001531 /* TerminalWindowPortal.swift */; };
|
||||
D0B10006A1B2C3D4E5F60001 /* TerminalWindowPortalDebug.swift in Sources */ = {isa = PBXBuildFile; fileRef = D0B10007A1B2C3D4E5F60001 /* TerminalWindowPortalDebug.swift */; };
|
||||
B7758A010000000000000001 /* TerminationWatchdog.swift in Sources */ = {isa = PBXBuildFile; fileRef = B7758A010000000000000002 /* TerminationWatchdog.swift */; };
|
||||
B7758A030000000000000001 /* TerminationWatchdogAtomic.c in Sources */ = {isa = PBXBuildFile; fileRef = B7758A030000000000000002 /* TerminationWatchdogAtomic.c */; };
|
||||
B7758A020000000000000001 /* TerminationWatchdogTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = B7758A020000000000000002 /* TerminationWatchdogTests.swift */; };
|
||||
C0DE7B100000000000000001 /* TextBoxInput.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0DE7B100000000000000002 /* TextBoxInput.swift */; };
|
||||
C0DE7B310000000000000001 /* TextBoxMentionCachedIndex.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0DE7B310000000000000002 /* TextBoxMentionCachedIndex.swift */; };
|
||||
C0DE7B250000000000000001 /* TextBoxMentionCandidate.swift in Sources */ = {isa = PBXBuildFile; fileRef = C0DE7B250000000000000002 /* TextBoxMentionCandidate.swift */; };
|
||||
@@ -2237,6 +2240,10 @@
|
||||
31B04B98376C9BA8B9E44DCB /* TerminalViewportUITestRecorder.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = TerminalViewportUITestRecorder.swift; sourceTree = "<group>"; };
|
||||
A5001531 /* TerminalWindowPortal.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TerminalWindowPortal.swift; sourceTree = "<group>"; };
|
||||
D0B10007A1B2C3D4E5F60001 /* TerminalWindowPortalDebug.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TerminalWindowPortalDebug.swift; sourceTree = "<group>"; };
|
||||
B7758A010000000000000002 /* TerminationWatchdog.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TerminationWatchdog.swift; sourceTree = "<group>"; };
|
||||
B7758A030000000000000002 /* TerminationWatchdogAtomic.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = TerminationWatchdogAtomic.c; sourceTree = "<group>"; };
|
||||
B7758A040000000000000002 /* TerminationWatchdogAtomic.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = TerminationWatchdogAtomic.h; sourceTree = "<group>"; };
|
||||
B7758A020000000000000002 /* TerminationWatchdogTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TerminationWatchdogTests.swift; sourceTree = "<group>"; };
|
||||
C0DE7B100000000000000002 /* TextBoxInput.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TextBoxInput.swift; sourceTree = "<group>"; };
|
||||
C0DE7B310000000000000002 /* TextBoxMentionCachedIndex.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TextBoxMentionCachedIndex.swift; sourceTree = "<group>"; };
|
||||
C0DE7B250000000000000002 /* TextBoxMentionCandidate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = TextBoxMentionCandidate.swift; sourceTree = "<group>"; };
|
||||
@@ -2957,6 +2964,9 @@
|
||||
A5001090 /* AppDelegate.swift */,
|
||||
C4A570010000000000000002 /* AppDelegate+CanvasShortcutRouting.swift */,
|
||||
A64610010000000000000002 /* QuitConfirmationAlertPresenter.swift */,
|
||||
B7758A030000000000000002 /* TerminationWatchdogAtomic.c */,
|
||||
B7758A040000000000000002 /* TerminationWatchdogAtomic.h */,
|
||||
B7758A010000000000000002 /* TerminationWatchdog.swift */,
|
||||
3865B0063865B0063865B006 /* AppDelegate+GlobalSearch.swift */,
|
||||
C3873002C3873002C3873002 /* GhosttyCrashBreadcrumb.swift */,
|
||||
C3873004C3873004C3873004 /* GhosttyCrashReportMetadata.swift */,
|
||||
@@ -3548,6 +3558,7 @@
|
||||
C0DE10510000000000000002 /* MobileHostServiceSettingsTests.swift */,
|
||||
6083A7DAD962E287FC2FFE94 /* ShortcutAndCommandPaletteTests.swift */,
|
||||
A64610020000000000000002 /* QuitConfirmationAlertPresenterTests.swift */,
|
||||
B7758A020000000000000002 /* TerminationWatchdogTests.swift */,
|
||||
6042B0016042B0016042B001 /* ShortcutHintModifierPolicyTests.swift */,
|
||||
C3408A000000000000000004 /* RightSidebarCommandPaletteTests.swift */,
|
||||
C0DE48000000000000000002 /* SidebarProviderMenuRegressionTests.swift */,
|
||||
@@ -4712,6 +4723,8 @@
|
||||
5154BEAB50364B86A9E36E4B /* TerminalViewportUITestRecorder.swift in Sources */,
|
||||
A5001532 /* TerminalWindowPortal.swift in Sources */,
|
||||
D0B10006A1B2C3D4E5F60001 /* TerminalWindowPortalDebug.swift in Sources */,
|
||||
B7758A010000000000000001 /* TerminationWatchdog.swift in Sources */,
|
||||
B7758A030000000000000001 /* TerminationWatchdogAtomic.c in Sources */,
|
||||
C0DE7B100000000000000001 /* TextBoxInput.swift in Sources */,
|
||||
C0DE7B310000000000000001 /* TextBoxMentionCachedIndex.swift in Sources */,
|
||||
C0DE7B250000000000000001 /* TextBoxMentionCandidate.swift in Sources */,
|
||||
@@ -5173,6 +5186,7 @@
|
||||
D5671101D5671101D5671101 /* TerminalScrollSpeedSettingsFileStoreTests.swift in Sources */,
|
||||
C0DE53360000000000000001 /* TerminalSearchOverlayMouseReleaseTests.swift in Sources */,
|
||||
D3052001000000000000001 /* TerminalSurfaceResizePolicyTests.swift in Sources */,
|
||||
B7758A020000000000000001 /* TerminationWatchdogTests.swift in Sources */,
|
||||
C0DE7B300000000000000001 /* TextBoxMentionCompletionTests.swift in Sources */,
|
||||
F50030040000000000000001 /* TitlebarInteractiveControlTests.swift in Sources */,
|
||||
D3284001A1B2C3D4E5F60718 /* TraditionalChineseIMENumpadRegressionTests.swift in Sources */,
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
import Foundation
|
||||
import Testing
|
||||
|
||||
#if canImport(cmux_DEV)
|
||||
@testable import cmux_DEV
|
||||
#elseif canImport(cmux)
|
||||
@testable import cmux
|
||||
#endif
|
||||
|
||||
@Suite
|
||||
struct TerminationWatchdogTests {
|
||||
/// Arming is idempotent: repeated calls (multiple quit attempts, or both the
|
||||
/// primary and backstop commit sites arming for one request) schedule the
|
||||
/// deadline exactly once, and nothing fires before the deadline elapses.
|
||||
///
|
||||
/// Deterministic by construction — the injected scheduler captures the
|
||||
/// deadline handler instead of sleeping, so no real time is involved
|
||||
/// (https://github.com/manaflow-ai/cmux/issues/6758).
|
||||
@Test
|
||||
func repeatedArmingSchedulesTheDeadlineExactlyOnce() {
|
||||
let scheduler = CapturingScheduler()
|
||||
let counter = FireCounter()
|
||||
let watchdog = TerminationWatchdog(
|
||||
onFire: { counter.increment() },
|
||||
scheduleDeadline: scheduler.schedule
|
||||
)
|
||||
|
||||
watchdog.arm(deadline: 8)
|
||||
watchdog.arm(deadline: 8)
|
||||
watchdog.arm(deadline: 8)
|
||||
|
||||
#expect(scheduler.scheduledCount == 1)
|
||||
#expect(counter.value == 0)
|
||||
}
|
||||
|
||||
/// When the deadline elapses, the watchdog runs its handler exactly once.
|
||||
@Test
|
||||
func elapsedDeadlineFiresTheHandlerExactlyOnce() {
|
||||
let scheduler = CapturingScheduler()
|
||||
let counter = FireCounter()
|
||||
let watchdog = TerminationWatchdog(
|
||||
onFire: { counter.increment() },
|
||||
scheduleDeadline: scheduler.schedule
|
||||
)
|
||||
|
||||
watchdog.arm(deadline: 8)
|
||||
scheduler.fireAll() // advance virtual time to the deadline
|
||||
|
||||
#expect(counter.value == 1)
|
||||
}
|
||||
|
||||
/// Captures scheduled deadline handlers instead of sleeping, so tests advance
|
||||
/// time by hand and stay deterministic.
|
||||
private final class CapturingScheduler: Sendable {
|
||||
private let lock = NSLock()
|
||||
// SAFETY: guarded by `lock`.
|
||||
nonisolated(unsafe) private var fires: [@Sendable () -> Void] = []
|
||||
|
||||
var schedule: TerminationWatchdog.DeadlineScheduler {
|
||||
{ [self] _, fire in
|
||||
lock.lock()
|
||||
fires.append(fire)
|
||||
lock.unlock()
|
||||
}
|
||||
}
|
||||
|
||||
var scheduledCount: Int {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
return fires.count
|
||||
}
|
||||
|
||||
func fireAll() {
|
||||
lock.lock()
|
||||
let snapshot = fires
|
||||
lock.unlock()
|
||||
for fire in snapshot { fire() }
|
||||
}
|
||||
}
|
||||
|
||||
private final class FireCounter: Sendable {
|
||||
private let lock = NSLock()
|
||||
// SAFETY: guarded by `lock`.
|
||||
nonisolated(unsafe) private var stored = 0
|
||||
|
||||
func increment() {
|
||||
lock.lock()
|
||||
stored += 1
|
||||
lock.unlock()
|
||||
}
|
||||
|
||||
var value: Int {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
return stored
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user