* docs: iOS browser streaming design * Add mobile browser streaming wire protocol * Add Mac mobile browser stream host * Fix stream session compile (nonisolated encoder init) and momentum end phase * Fix keyCode method shadowing in SyntheticKeyEventFactory * Add iOS browser stream surface package * Add browser stream RPC client plumbing * Wire browser streams through the mobile shell * Integrate browser streams into mobile shell UI * Beacon: detect canvas/WebGL painting via requestAnimationFrame wrap * ci: reload-build gains an ios-simulator platform Builds the unsigned simulator .app and uploads it as an artifact, for callers whose local xcodebuild is unavailable; the sim bundle installs directly via simctl. * ci: build the ios-simulator app arm64-only GhosttyKit's simulator slice is arm64-only, so the generic destination's x86_64 half fails at link; every target simulator is arm64. * Fix display link teardown for Swift 6 nonisolated deinit * Fix frame stall via store-owned decode pipeline; move chrome to bottom floating bar * Self-heal browser stream: force restart past dedupe on recovery, unanswered-input watchdog, keyboard-pinned bottom bar * Add mobile browser dialog wire model and broker * Mirror Mac browser dialogs over mobile RPC * Render mirrored browser dialogs on iOS * Wire mobile browser dialog Mac sources into Xcode project * Capture owner explicitly in basic-auth startPrompt closure * Stack browser dialog buttons vertically for 3+ or long labels * Reserve bottom bar space so chrome never occludes streamed page content * Take main's reconnect route-isolation test (recoveryTask removed by Iroh fix) * Browser bar: always-visible standard controls, drop collapse pill + confusing X/chevron; stop stream on surface exit * Add mobile browser viewport RPC DTOs * Reflow Mac browser streams to phone viewport * iOS: report phone viewport to reflow the streamed Mac browser * Fix streamed browser white-out: force repaint after viewport reflow so idle pages don't capture a blank frame * White-out fix v2: real two-frame scroll repaint nudge + settle-capture burst after reflow * Replace iOS tab switcher surface * Fix iOS switcher integration and verification * Test persistent browser render host portal ownership * Share persistent browser offscreen render hosting * Capture mobile browser streams in persistent render host * Fix switcher initial positioning and accessibility * Test switcher reopening after browser selection * Reset switcher state for each presentation * iOS browser stream: mirror phone frames in the Mac pane instead of blanking it While a browser pane streams to the phone, the live WKWebView renders in the offscreen host at phone width, so the Mac pane went fully blank. Show a read-only, letterboxed, click-through mirror of the exact frames the phone receives (fed from the same capture in MobileBrowserStreamSession at the same cadence), added to the pane's superview on stream start and removed on teardown when the full-width live web view returns. Co-Authored-By: Claude Opus 4.8 <[email protected]> * Speed up browser stream capture on the offscreen render host Continuous JPEG frames were snapshotting with afterScreenUpdates:true, which blocks each takeSnapshot on the host window's screen-update cycle. The stream's offscreen render host lives off all screens at alpha ~0, where macOS throttles that cycle hard, so capture was capped to a few fps: the phone showed "super slow" streaming that barely moved on scroll. Snapshot continuous JPEG frames with afterScreenUpdates:false instead. That captures the currently committed render, which already reflects the new scroll offset, without waiting on the throttled cycle; the dirty loop re-captures to stay current. The rare lossless PNG settle frame keeps afterScreenUpdates:true for a pixel-perfect rest state. Add DEBUG per-capture instrumentation (capture ms, encode ms, byte size, pixel size, unacked count) so stream throughput is measurable from the debug log and capture-bound vs flow-controlled is distinguishable. Co-Authored-By: Claude Opus 4.8 <[email protected]> * Revert "Reset switcher state for each presentation" This reverts commited9d5f8b46. * Revert "Test switcher reopening after browser selection" This reverts commit2bfebb3346. * Revert "Fix switcher initial positioning and accessibility" This reverts commit607ef33924. * Revert "Fix iOS switcher integration and verification" This reverts commit9cfb181750. * Revert "Replace iOS tab switcher surface" This reverts commit89105d342d. * Revert "ci: build the ios-simulator app arm64-only" This reverts commitf5e9324940. * Revert "ci: reload-build gains an ios-simulator platform" This reverts commit42b65b2300. * Scope PR to browser streaming: drop switcher residue from title menu and string catalog Co-Authored-By: Claude Fable 5 <[email protected]> * Test replayed browser input requests a stream capture * Keep the stream render host visible to WebKit: on-screen floating window, input-replay dirty, event-driven scroll beacon The persistent render host window sat at (-100000,-100000); AppKit reports a window with no on-screen portion as fully occluded, and WebKit suspends requestAnimationFrame and degrades trusted-event hit testing for occluded hosts. The rAF-throttled dirty beacon therefore never fired during a scroll gesture (one frame per gesture, captured after gesture end) and replayed taps intermittently hit a stale tree and never navigated. Host window now anchors on-screen (bottom-trailing, >=64pt visible, .floating so ordinary windows cannot occlude it) while staying imperceptible (1% alpha, click-through, non-activating). Hardening: every replayed input batch marks the session dirty directly, and the beacon posts scroll/wheel dirt from the event listener with a 16ms throttle instead of waiting for a rAF tick. Co-Authored-By: Claude Fable 5 <[email protected]> * Bind the browser-stream keyboard button to real keyboard visibility The button showed the input proxy's focus intent, so a keyboard raised by the address field or a dialog's text field left it stuck on 'Show Keyboard'. The glyph now binds to MobileKeyboardVisibilityObserver (UIKit keyboard notifications); tapping while the keyboard is up resigns whichever responder raised it (shared dismissMobileKeyboard, moved to CmuxMobileSupport) and releases the proxy's focus reasons via the policy's new explicit hide, which never flips into a focus request the way toggling would. Co-Authored-By: Claude Fable 5 <[email protected]> * Give dialog text fields a visible input well The dialog card is glass, so the fields' glass background vanished into it and prompt/basic-auth inputs read as labels. Fields now sit in a filled rounded well with a hairline border, the same fill language as the bottom bar's address field. Co-Authored-By: Claude Fable 5 <[email protected]> --------- Co-authored-by: cmux reload-cloud <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]>
220 lines
7.5 KiB
Swift
220 lines
7.5 KiB
Swift
import AppKit
|
|
import CoreGraphics
|
|
import Foundation
|
|
|
|
struct SyntheticKeySpecification {
|
|
let storedKey: String
|
|
let keyCode: UInt16
|
|
let modifierFlags: NSEvent.ModifierFlags
|
|
let characters: String
|
|
let charactersIgnoringModifiers: String
|
|
}
|
|
|
|
enum SyntheticKeyEventFactory {
|
|
static func parseShortcutCombo(_ combo: String) -> SyntheticKeySpecification? {
|
|
let raw = combo.trimmingCharacters(in: .whitespacesAndNewlines)
|
|
guard !raw.isEmpty else { return nil }
|
|
|
|
let parts = raw
|
|
.split(separator: "+")
|
|
.map { String($0).trimmingCharacters(in: .whitespacesAndNewlines) }
|
|
.filter { !$0.isEmpty }
|
|
guard !parts.isEmpty else { return nil }
|
|
|
|
var flags: NSEvent.ModifierFlags = []
|
|
var keyToken: String?
|
|
for part in parts {
|
|
if let flag = modifierFlag(named: part) {
|
|
flags.insert(flag)
|
|
} else if keyToken == nil {
|
|
keyToken = part
|
|
} else {
|
|
return nil
|
|
}
|
|
}
|
|
guard let keyToken else { return nil }
|
|
return specification(key: keyToken, modifierFlags: flags)
|
|
}
|
|
|
|
static func specification(
|
|
key: String,
|
|
modifierNames: [String]
|
|
) -> SyntheticKeySpecification? {
|
|
var flags: NSEvent.ModifierFlags = []
|
|
for name in modifierNames {
|
|
guard let flag = modifierFlag(named: name) else { return nil }
|
|
flags.insert(flag)
|
|
}
|
|
return specification(key: key, modifierFlags: flags)
|
|
}
|
|
|
|
static func specification(forASCIICharacter character: Character) -> SyntheticKeySpecification? {
|
|
switch character {
|
|
case "\n", "\r": return specification(key: "return", modifierFlags: [])
|
|
case "\t": return specification(key: "tab", modifierFlags: [])
|
|
case " ": return specification(key: "space", modifierFlags: [])
|
|
default: break
|
|
}
|
|
|
|
let shiftedSymbols: [Character: Character] = [
|
|
"!": "1", "@": "2", "#": "3", "$": "4", "%": "5", "^": "6", "&": "7", "*": "8", "(": "9", ")": "0",
|
|
"_": "-", "+": "=", "{": "[", "}": "]", "|": "\\", ":": ";", "\"": "'", "<": ",", ">": ".", "?": "/", "~": "`",
|
|
]
|
|
let requiresShift = character.isUppercase || shiftedSymbols[character] != nil
|
|
let base = shiftedSymbols[character].map(String.init) ?? String(character).lowercased()
|
|
guard var result = specification(
|
|
key: base,
|
|
modifierFlags: requiresShift ? [.shift] : []
|
|
) else { return nil }
|
|
result = SyntheticKeySpecification(
|
|
storedKey: result.storedKey,
|
|
keyCode: result.keyCode,
|
|
modifierFlags: result.modifierFlags,
|
|
characters: String(character),
|
|
charactersIgnoringModifiers: String(character)
|
|
)
|
|
return result
|
|
}
|
|
|
|
/// Builds an `NSEvent` backed by a real `CGEvent` so WebKit text input can
|
|
/// safely interpret it. Callers choose their own direct delivery target.
|
|
static func keyEvent(
|
|
specification: SyntheticKeySpecification,
|
|
keyDown: Bool,
|
|
timestamp: TimeInterval,
|
|
characters: String? = nil
|
|
) -> NSEvent? {
|
|
guard let cgEvent = CGEvent(
|
|
keyboardEventSource: nil,
|
|
virtualKey: specification.keyCode,
|
|
keyDown: keyDown
|
|
) else { return nil }
|
|
cgEvent.flags = cgFlags(specification.modifierFlags)
|
|
cgEvent.timestamp = CGEventTimestamp(timestamp * 1_000_000_000)
|
|
if let characters {
|
|
var utf16 = Array(characters.utf16)
|
|
cgEvent.keyboardSetUnicodeString(stringLength: utf16.count, unicodeString: &utf16)
|
|
}
|
|
return NSEvent(cgEvent: cgEvent)
|
|
}
|
|
|
|
private static func specification(
|
|
key rawKey: String,
|
|
modifierFlags: NSEvent.ModifierFlags
|
|
) -> SyntheticKeySpecification? {
|
|
let key = rawKey.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
|
|
guard !key.isEmpty else { return nil }
|
|
|
|
let storedKey: String
|
|
let keyCode: UInt16
|
|
let charactersIgnoringModifiers: String
|
|
switch key {
|
|
case "left": (storedKey, keyCode) = ("\u{F702}", 123)
|
|
case "right": (storedKey, keyCode) = ("\u{F703}", 124)
|
|
case "down": (storedKey, keyCode) = ("\u{F701}", 125)
|
|
case "up": (storedKey, keyCode) = ("\u{F700}", 126)
|
|
case "enter", "return": (storedKey, keyCode) = ("\r", 36)
|
|
case "tab": (storedKey, keyCode) = ("\t", 48)
|
|
case "escape", "esc": (storedKey, keyCode) = ("\u{1b}", 53)
|
|
case "delete", "backspace": (storedKey, keyCode) = ("\u{8}", 51)
|
|
case "forward_delete": (storedKey, keyCode) = ("\u{f728}", 117)
|
|
case "space": (storedKey, keyCode) = (" ", 49)
|
|
default:
|
|
guard key.count == 1, let resolved = Self.keyCode(for: key) else { return nil }
|
|
storedKey = key
|
|
keyCode = resolved
|
|
}
|
|
|
|
if modifierFlags.contains(.control),
|
|
storedKey.count == 1,
|
|
let scalar = storedKey.unicodeScalars.first,
|
|
scalar.isASCII,
|
|
scalar.value >= 97, scalar.value <= 122 {
|
|
charactersIgnoringModifiers = String(UnicodeScalar(scalar.value - 96)!)
|
|
} else {
|
|
charactersIgnoringModifiers = storedKey
|
|
}
|
|
return SyntheticKeySpecification(
|
|
storedKey: storedKey,
|
|
keyCode: keyCode,
|
|
modifierFlags: modifierFlags,
|
|
characters: charactersIgnoringModifiers,
|
|
charactersIgnoringModifiers: charactersIgnoringModifiers
|
|
)
|
|
}
|
|
|
|
private static func modifierFlag(named rawName: String) -> NSEvent.ModifierFlags? {
|
|
switch rawName.lowercased() {
|
|
case "cmd", "command", "super": return .command
|
|
case "ctrl", "control": return .control
|
|
case "opt", "option", "alt": return .option
|
|
case "shift": return .shift
|
|
default: return nil
|
|
}
|
|
}
|
|
|
|
private static func cgFlags(_ modifiers: NSEvent.ModifierFlags) -> CGEventFlags {
|
|
var flags: CGEventFlags = []
|
|
if modifiers.contains(.command) { flags.insert(.maskCommand) }
|
|
if modifiers.contains(.control) { flags.insert(.maskControl) }
|
|
if modifiers.contains(.option) { flags.insert(.maskAlternate) }
|
|
if modifiers.contains(.shift) { flags.insert(.maskShift) }
|
|
if modifiers.contains(.capsLock) { flags.insert(.maskAlphaShift) }
|
|
if modifiers.contains(.function) { flags.insert(.maskSecondaryFn) }
|
|
return flags
|
|
}
|
|
|
|
private static func keyCode(for key: String) -> UInt16? {
|
|
switch key {
|
|
case "a": 0
|
|
case "s": 1
|
|
case "d": 2
|
|
case "f": 3
|
|
case "h": 4
|
|
case "g": 5
|
|
case "z": 6
|
|
case "x": 7
|
|
case "c": 8
|
|
case "v": 9
|
|
case "b": 11
|
|
case "q": 12
|
|
case "w": 13
|
|
case "e": 14
|
|
case "r": 15
|
|
case "y": 16
|
|
case "t": 17
|
|
case "1": 18
|
|
case "2": 19
|
|
case "3": 20
|
|
case "4": 21
|
|
case "6": 22
|
|
case "5": 23
|
|
case "=": 24
|
|
case "9": 25
|
|
case "7": 26
|
|
case "-": 27
|
|
case "8": 28
|
|
case "0": 29
|
|
case "]": 30
|
|
case "o": 31
|
|
case "u": 32
|
|
case "[": 33
|
|
case "i": 34
|
|
case "p": 35
|
|
case "l": 37
|
|
case "j": 38
|
|
case "'": 39
|
|
case "k": 40
|
|
case ";": 41
|
|
case "\\": 42
|
|
case ",": 43
|
|
case "/": 44
|
|
case "n": 45
|
|
case "m": 46
|
|
case ".": 47
|
|
case "`": 50
|
|
default: nil
|
|
}
|
|
}
|
|
}
|