Files
cmux/Sources/SyntheticKeyEventFactory.swift
b86cce2e7d iOS: stream Mac browser panes to the phone (pixel-perfect, interactive, dialogs mirrored) (#8298)
* docs: iOS browser streaming design

* Add mobile browser streaming wire protocol

* Add Mac mobile browser stream host

* Fix stream session compile (nonisolated encoder init) and momentum end phase

* Fix keyCode method shadowing in SyntheticKeyEventFactory

* Add iOS browser stream surface package

* Add browser stream RPC client plumbing

* Wire browser streams through the mobile shell

* Integrate browser streams into mobile shell UI

* Beacon: detect canvas/WebGL painting via requestAnimationFrame wrap

* ci: reload-build gains an ios-simulator platform

Builds the unsigned simulator .app and uploads it as an artifact, for
callers whose local xcodebuild is unavailable; the sim bundle installs
directly via simctl.

* ci: build the ios-simulator app arm64-only

GhosttyKit's simulator slice is arm64-only, so the generic destination's
x86_64 half fails at link; every target simulator is arm64.

* Fix display link teardown for Swift 6 nonisolated deinit

* Fix frame stall via store-owned decode pipeline; move chrome to bottom floating bar

* Self-heal browser stream: force restart past dedupe on recovery, unanswered-input watchdog, keyboard-pinned bottom bar

* Add mobile browser dialog wire model and broker

* Mirror Mac browser dialogs over mobile RPC

* Render mirrored browser dialogs on iOS

* Wire mobile browser dialog Mac sources into Xcode project

* Capture owner explicitly in basic-auth startPrompt closure

* Stack browser dialog buttons vertically for 3+ or long labels

* Reserve bottom bar space so chrome never occludes streamed page content

* Take main's reconnect route-isolation test (recoveryTask removed by Iroh fix)

* Browser bar: always-visible standard controls, drop collapse pill + confusing X/chevron; stop stream on surface exit

* Add mobile browser viewport RPC DTOs

* Reflow Mac browser streams to phone viewport

* iOS: report phone viewport to reflow the streamed Mac browser

* Fix streamed browser white-out: force repaint after viewport reflow so idle pages don't capture a blank frame

* White-out fix v2: real two-frame scroll repaint nudge + settle-capture burst after reflow

* Replace iOS tab switcher surface

* Fix iOS switcher integration and verification

* Test persistent browser render host portal ownership

* Share persistent browser offscreen render hosting

* Capture mobile browser streams in persistent render host

* Fix switcher initial positioning and accessibility

* Test switcher reopening after browser selection

* Reset switcher state for each presentation

* iOS browser stream: mirror phone frames in the Mac pane instead of blanking it

While a browser pane streams to the phone, the live WKWebView renders in the
offscreen host at phone width, so the Mac pane went fully blank. Show a
read-only, letterboxed, click-through mirror of the exact frames the phone
receives (fed from the same capture in MobileBrowserStreamSession at the same
cadence), added to the pane's superview on stream start and removed on teardown
when the full-width live web view returns.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* Speed up browser stream capture on the offscreen render host

Continuous JPEG frames were snapshotting with afterScreenUpdates:true, which
blocks each takeSnapshot on the host window's screen-update cycle. The stream's
offscreen render host lives off all screens at alpha ~0, where macOS throttles
that cycle hard, so capture was capped to a few fps: the phone showed "super
slow" streaming that barely moved on scroll.

Snapshot continuous JPEG frames with afterScreenUpdates:false instead. That
captures the currently committed render, which already reflects the new scroll
offset, without waiting on the throttled cycle; the dirty loop re-captures to
stay current. The rare lossless PNG settle frame keeps afterScreenUpdates:true
for a pixel-perfect rest state.

Add DEBUG per-capture instrumentation (capture ms, encode ms, byte size, pixel
size, unacked count) so stream throughput is measurable from the debug log and
capture-bound vs flow-controlled is distinguishable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* Revert "Reset switcher state for each presentation"

This reverts commit ed9d5f8b46.

* Revert "Test switcher reopening after browser selection"

This reverts commit 2bfebb3346.

* Revert "Fix switcher initial positioning and accessibility"

This reverts commit 607ef33924.

* Revert "Fix iOS switcher integration and verification"

This reverts commit 9cfb181750.

* Revert "Replace iOS tab switcher surface"

This reverts commit 89105d342d.

* Revert "ci: build the ios-simulator app arm64-only"

This reverts commit f5e9324940.

* Revert "ci: reload-build gains an ios-simulator platform"

This reverts commit 42b65b2300.

* Scope PR to browser streaming: drop switcher residue from title menu and string catalog

Co-Authored-By: Claude Fable 5 <[email protected]>

* Test replayed browser input requests a stream capture

* Keep the stream render host visible to WebKit: on-screen floating window, input-replay dirty, event-driven scroll beacon

The persistent render host window sat at (-100000,-100000); AppKit reports a
window with no on-screen portion as fully occluded, and WebKit suspends
requestAnimationFrame and degrades trusted-event hit testing for occluded
hosts. The rAF-throttled dirty beacon therefore never fired during a scroll
gesture (one frame per gesture, captured after gesture end) and replayed taps
intermittently hit a stale tree and never navigated.

Host window now anchors on-screen (bottom-trailing, >=64pt visible, .floating
so ordinary windows cannot occlude it) while staying imperceptible (1% alpha,
click-through, non-activating). Hardening: every replayed input batch marks
the session dirty directly, and the beacon posts scroll/wheel dirt from the
event listener with a 16ms throttle instead of waiting for a rAF tick.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Bind the browser-stream keyboard button to real keyboard visibility

The button showed the input proxy's focus intent, so a keyboard raised by the
address field or a dialog's text field left it stuck on 'Show Keyboard'.
The glyph now binds to MobileKeyboardVisibilityObserver (UIKit keyboard
notifications); tapping while the keyboard is up resigns whichever responder
raised it (shared dismissMobileKeyboard, moved to CmuxMobileSupport) and
releases the proxy's focus reasons via the policy's new explicit hide, which
never flips into a focus request the way toggling would.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Give dialog text fields a visible input well

The dialog card is glass, so the fields' glass background vanished into it and
prompt/basic-auth inputs read as labels. Fields now sit in a filled rounded
well with a hairline border, the same fill language as the bottom bar's
address field.

Co-Authored-By: Claude Fable 5 <[email protected]>

---------

Co-authored-by: cmux reload-cloud <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
2026-07-30 16:13:11 -05:00

220 lines
7.5 KiB
Swift

import AppKit
import CoreGraphics
import Foundation
struct SyntheticKeySpecification {
let storedKey: String
let keyCode: UInt16
let modifierFlags: NSEvent.ModifierFlags
let characters: String
let charactersIgnoringModifiers: String
}
enum SyntheticKeyEventFactory {
static func parseShortcutCombo(_ combo: String) -> SyntheticKeySpecification? {
let raw = combo.trimmingCharacters(in: .whitespacesAndNewlines)
guard !raw.isEmpty else { return nil }
let parts = raw
.split(separator: "+")
.map { String($0).trimmingCharacters(in: .whitespacesAndNewlines) }
.filter { !$0.isEmpty }
guard !parts.isEmpty else { return nil }
var flags: NSEvent.ModifierFlags = []
var keyToken: String?
for part in parts {
if let flag = modifierFlag(named: part) {
flags.insert(flag)
} else if keyToken == nil {
keyToken = part
} else {
return nil
}
}
guard let keyToken else { return nil }
return specification(key: keyToken, modifierFlags: flags)
}
static func specification(
key: String,
modifierNames: [String]
) -> SyntheticKeySpecification? {
var flags: NSEvent.ModifierFlags = []
for name in modifierNames {
guard let flag = modifierFlag(named: name) else { return nil }
flags.insert(flag)
}
return specification(key: key, modifierFlags: flags)
}
static func specification(forASCIICharacter character: Character) -> SyntheticKeySpecification? {
switch character {
case "\n", "\r": return specification(key: "return", modifierFlags: [])
case "\t": return specification(key: "tab", modifierFlags: [])
case " ": return specification(key: "space", modifierFlags: [])
default: break
}
let shiftedSymbols: [Character: Character] = [
"!": "1", "@": "2", "#": "3", "$": "4", "%": "5", "^": "6", "&": "7", "*": "8", "(": "9", ")": "0",
"_": "-", "+": "=", "{": "[", "}": "]", "|": "\\", ":": ";", "\"": "'", "<": ",", ">": ".", "?": "/", "~": "`",
]
let requiresShift = character.isUppercase || shiftedSymbols[character] != nil
let base = shiftedSymbols[character].map(String.init) ?? String(character).lowercased()
guard var result = specification(
key: base,
modifierFlags: requiresShift ? [.shift] : []
) else { return nil }
result = SyntheticKeySpecification(
storedKey: result.storedKey,
keyCode: result.keyCode,
modifierFlags: result.modifierFlags,
characters: String(character),
charactersIgnoringModifiers: String(character)
)
return result
}
/// Builds an `NSEvent` backed by a real `CGEvent` so WebKit text input can
/// safely interpret it. Callers choose their own direct delivery target.
static func keyEvent(
specification: SyntheticKeySpecification,
keyDown: Bool,
timestamp: TimeInterval,
characters: String? = nil
) -> NSEvent? {
guard let cgEvent = CGEvent(
keyboardEventSource: nil,
virtualKey: specification.keyCode,
keyDown: keyDown
) else { return nil }
cgEvent.flags = cgFlags(specification.modifierFlags)
cgEvent.timestamp = CGEventTimestamp(timestamp * 1_000_000_000)
if let characters {
var utf16 = Array(characters.utf16)
cgEvent.keyboardSetUnicodeString(stringLength: utf16.count, unicodeString: &utf16)
}
return NSEvent(cgEvent: cgEvent)
}
private static func specification(
key rawKey: String,
modifierFlags: NSEvent.ModifierFlags
) -> SyntheticKeySpecification? {
let key = rawKey.trimmingCharacters(in: .whitespacesAndNewlines).lowercased()
guard !key.isEmpty else { return nil }
let storedKey: String
let keyCode: UInt16
let charactersIgnoringModifiers: String
switch key {
case "left": (storedKey, keyCode) = ("\u{F702}", 123)
case "right": (storedKey, keyCode) = ("\u{F703}", 124)
case "down": (storedKey, keyCode) = ("\u{F701}", 125)
case "up": (storedKey, keyCode) = ("\u{F700}", 126)
case "enter", "return": (storedKey, keyCode) = ("\r", 36)
case "tab": (storedKey, keyCode) = ("\t", 48)
case "escape", "esc": (storedKey, keyCode) = ("\u{1b}", 53)
case "delete", "backspace": (storedKey, keyCode) = ("\u{8}", 51)
case "forward_delete": (storedKey, keyCode) = ("\u{f728}", 117)
case "space": (storedKey, keyCode) = (" ", 49)
default:
guard key.count == 1, let resolved = Self.keyCode(for: key) else { return nil }
storedKey = key
keyCode = resolved
}
if modifierFlags.contains(.control),
storedKey.count == 1,
let scalar = storedKey.unicodeScalars.first,
scalar.isASCII,
scalar.value >= 97, scalar.value <= 122 {
charactersIgnoringModifiers = String(UnicodeScalar(scalar.value - 96)!)
} else {
charactersIgnoringModifiers = storedKey
}
return SyntheticKeySpecification(
storedKey: storedKey,
keyCode: keyCode,
modifierFlags: modifierFlags,
characters: charactersIgnoringModifiers,
charactersIgnoringModifiers: charactersIgnoringModifiers
)
}
private static func modifierFlag(named rawName: String) -> NSEvent.ModifierFlags? {
switch rawName.lowercased() {
case "cmd", "command", "super": return .command
case "ctrl", "control": return .control
case "opt", "option", "alt": return .option
case "shift": return .shift
default: return nil
}
}
private static func cgFlags(_ modifiers: NSEvent.ModifierFlags) -> CGEventFlags {
var flags: CGEventFlags = []
if modifiers.contains(.command) { flags.insert(.maskCommand) }
if modifiers.contains(.control) { flags.insert(.maskControl) }
if modifiers.contains(.option) { flags.insert(.maskAlternate) }
if modifiers.contains(.shift) { flags.insert(.maskShift) }
if modifiers.contains(.capsLock) { flags.insert(.maskAlphaShift) }
if modifiers.contains(.function) { flags.insert(.maskSecondaryFn) }
return flags
}
private static func keyCode(for key: String) -> UInt16? {
switch key {
case "a": 0
case "s": 1
case "d": 2
case "f": 3
case "h": 4
case "g": 5
case "z": 6
case "x": 7
case "c": 8
case "v": 9
case "b": 11
case "q": 12
case "w": 13
case "e": 14
case "r": 15
case "y": 16
case "t": 17
case "1": 18
case "2": 19
case "3": 20
case "4": 21
case "6": 22
case "5": 23
case "=": 24
case "9": 25
case "7": 26
case "-": 27
case "8": 28
case "0": 29
case "]": 30
case "o": 31
case "u": 32
case "[": 33
case "i": 34
case "p": 35
case "l": 37
case "j": 38
case "'": 39
case "k": 40
case ";": 41
case "\\": 42
case ",": 43
case "/": 44
case "n": 45
case "m": 46
case ".": 47
case "`": 50
default: nil
}
}
}