ios: isolate Agent input and read observation
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
# Agent scrolling benchmark
|
||||
# Agent interaction benchmarks
|
||||
|
||||
Run the production Agent view in an app-hosted XCTest on the iPhone 16e simulator:
|
||||
|
||||
@@ -7,62 +7,88 @@ just generate
|
||||
xcodebuild test -scheme SybilPresentation \
|
||||
-destination 'platform=iOS Simulator,name=iPhone 16e,OS=latest' \
|
||||
-derivedDataPath build/DerivedData -parallel-testing-enabled NO \
|
||||
-only-testing:SybilPresentationTests/AgentPerformanceTests \
|
||||
-resultBundlePath build/agent-performance.xcresult
|
||||
-only-testing:SybilPresentationTests/AgentPerformanceTests/testInputAndEntranceResponsiveness \
|
||||
-resultBundlePath build/agent-input-performance.xcresult
|
||||
```
|
||||
|
||||
`AgentPerformanceTests` seeds a room through its normal event reducer with 160
|
||||
local messages (100 initially displayed). User text is literal; assistant text
|
||||
contains paragraphs, lists, links, emphasis and code. The real Agent view,
|
||||
composer, avatar, entrance layout and thinking spinner remain enabled. No server,
|
||||
production credentials, notification delivery or network images are needed.
|
||||
`testInputAndEntranceResponsiveness` seeds 160 local mixed literal/Markdown
|
||||
messages through the normal room reducer (100 initially displayed), then:
|
||||
|
||||
Three approximately 12-second phases run in order: idle scrolling, thinking at
|
||||
the bottom, and thinking while scrolling. The test explicitly resets the offset
|
||||
to the bottom before the stationary thinking phase (including in the baseline).
|
||||
Thinking activity changes queue depth, worker count and tool progress every
|
||||
500 ms through the normal 100 ms publication
|
||||
batch. The display link sweeps a 2400-point range at 600 points/second, reversing
|
||||
every four seconds. Use the **same test source, simulator, build configuration
|
||||
and host load** for before/after runs. Don't run other builds or profilers during
|
||||
a measured run. Warm builds and repeat runs help distinguish stable effects from
|
||||
host noise.
|
||||
- Receives 12 more messages at 550 ms intervals through the production 100 ms
|
||||
publication batch and 240 ms entrance animation, checking bottom following.
|
||||
- Presents and dismisses the actual software keyboard six times, 600 ms per
|
||||
transition. It asserts that the transcript viewport shrinks by over 100 points
|
||||
and remains at the bottom; a hardware-keyboard-only run is invalid.
|
||||
- Calls UIKit `insertText` on the actual focused composer every 30 ms, including
|
||||
enough text to wrap and grow to its six-line limit. It verifies the draft,
|
||||
bottom following, and that keyboard/draft changes don't reproject the history.
|
||||
|
||||
The geometry assertions use iPhone 16e, iOS 26.5, and the default text size. They
|
||||
retain the unoptimized transcript heights (39181.333 points idle, 39223.333 while
|
||||
thinking), so an isolation wrapper cannot silently change row spacing. Rebaseline
|
||||
these explicitly when changing simulator/font configuration or the fixture.
|
||||
The real composer, Markdown rows, avatars and entrance animations remain enabled.
|
||||
No server, production credentials, notification delivery or network images are
|
||||
needed. The log prints `AGENT_INPUT_PERFORMANCE_JSON`; the result bundle retains
|
||||
JSON and screenshots of the actual hosted view before and after typing.
|
||||
Screenshots capture the hosting view, not the separate system keyboard window.
|
||||
|
||||
The log prints `AGENT_PERFORMANCE_JSON`; the result bundle retains the JSON as an
|
||||
attachment. Delivery-gap p50/p95/p99/max and counts over 25/50 ms measure actual
|
||||
main-run-loop responsiveness. `missedBudgetIntervals` estimates gaps relative to
|
||||
the reported display-link budget. These are **not compositor FPS, GPU hitches,
|
||||
touch latency or physical-device results**. Scrolling is programmatic; native
|
||||
swipe/anchor/read behavior needs the local-fixture UI regression tests as well.
|
||||
The hosted view excludes the surrounding app shell. Physical iPhone scrolling,
|
||||
streaming and thinking animation remain a required follow-up.
|
||||
Use the same test source, simulator, build configuration and host load for
|
||||
before/after measurements. Don't run other builds or profilers during sampling.
|
||||
Functional assertions about observation boundaries should fail against the old
|
||||
implementation; use the same sampling procedure when collecting a baseline.
|
||||
|
||||
Opt-in `AgentRenderDiagnostics` counters are compiled only in Debug and are off
|
||||
outside this test. They explain work (Agent body evaluations, message projection,
|
||||
message text construction, isolated row evaluations, geometry callbacks and
|
||||
snapshot publications). A drop in these counts alone does not establish improved
|
||||
frame responsiveness.
|
||||
## Measurement limits
|
||||
|
||||
## Reference result
|
||||
Delivery-gap p50/p95/p99/max and counts over 25/50 ms measure **CADisplayLink
|
||||
main-run-loop callback delivery**, not compositor FPS, GPU hitches, touch-to-key
|
||||
latency or physical-device performance. The hosted view excludes the surrounding
|
||||
app shell. UIKit input exercises editing/layout without synthesizing key taps.
|
||||
The first keyboard presentation includes system warmup; residual system and
|
||||
simulator scheduling gaps remain. Native gestures, read settling, header fade,
|
||||
attachments and reading anchors are covered separately by the loopback UI tests.
|
||||
|
||||
On monsoon, Xcode 26.5, Debug, iPhone 16e / iOS 26.5, clean repeat runs against
|
||||
`f534f76` plus counters and against the geometry/row isolation fix gave:
|
||||
`AgentRenderDiagnostics` counters are Debug-only and opt-in. They explain work;
|
||||
a drop in counts alone does not establish improved responsiveness.
|
||||
|
||||
| Phase | Before p95 / p99 (ms) | After p95 / p99 (ms) | Gaps >50 ms, before / after |
|
||||
The shipped transcript retains its eager stack and exact row geometry. A lazy
|
||||
experiment reduced measured work but failed the existing 2–3 point prepend
|
||||
anchor checks. Retaining a pending anchor across geometry updates also failed;
|
||||
an indefinite correction loop is not acceptable. The eager fallback keeps the
|
||||
existing one-shot correction, isolates composer/read observation, and removes
|
||||
row frame observers while following the bottom.
|
||||
|
||||
The experimental lazy content height changed from 43869 to 35870 points because
|
||||
unmaterialized rows use estimates, not because content was shortened. Its
|
||||
after-arrival screenshot was byte-identical to the eager baseline. Row text,
|
||||
spacing, attachments and entrance growth were unchanged; the anchor failure
|
||||
was sufficient to reject the lazy variant.
|
||||
|
||||
## Scrolling harness
|
||||
|
||||
`testThinkingScrollResponsiveness` retains the earlier three approximately
|
||||
12-second phases: idle scroll, stationary thinking, and scrolling with changing
|
||||
queue/worker/tool activity every 500 ms. Its display link sweeps 2400 points at
|
||||
600 points/second. This programmatic sweep does not reproduce native drag phases;
|
||||
use the UI tests for reading-anchor geometry and interactive keyboard dismissal.
|
||||
|
||||
The eager-stack reference in `b9a6f5d` has exact heights of 39181.333
|
||||
points idle and 39223.333 while thinking, retained as regression assertions. That earlier scrolling fix reduced busy-scroll p95 from 88.8 to
|
||||
19.4 ms; it did not address the input/entrance paths measured here. Physical
|
||||
phone validation remains necessary for hardware performance claims.
|
||||
|
||||
## 2026-10-16 focused comparison
|
||||
|
||||
Single Debug simulator runs on the same iPhone 16e / iOS 26.5, starting at
|
||||
`7510603` (which includes `b9a6f5d`). Values are callback-gap p95 milliseconds:
|
||||
|
||||
| Path | Before | Shipped eager isolation | Rejected lazy isolation |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Idle scroll | 50.01 / 50.02 | 16.68 / 16.71 | 128 / 0 |
|
||||
| Thinking stationary | 33.33 / 81.92 | 28.91 / 38.35 | 22 / 2 |
|
||||
| Thinking scroll | 88.78 / 90.89 | 19.41 / 30.42 | 123 / 1 |
|
||||
| 12 message entrances | 40.8 | 33.2 | 16.8 |
|
||||
| Six keyboard cycles | 33.2 | 33.1 | 29.2 |
|
||||
| Multiline composer typing | 24.8 | 25.2 | 21.3 |
|
||||
|
||||
Each phase lasted approximately 12 seconds. Content heights matched exactly.
|
||||
Busy-scroll Agent body/projection evaluations fell from 261 to 27; unchanged
|
||||
text constructions were zero in both versions. Geometry isolation alone removed
|
||||
the idle-scroll stalls; individual row isolation reduced activity-related gaps.
|
||||
These results support those two boundaries, without a Markdown cache or lazy
|
||||
layout change. Occasional simulator gaps remain; this does not replace device
|
||||
validation.
|
||||
Final transcript projections were 12/0/0 versus 220/24/10, with no row-frame
|
||||
preference callbacks in these pinned-bottom phases. Eager content height stayed
|
||||
43869.333 points. Entrance p99 improved 83.4→57.1 ms; keyboard p99 55.9→44.5;
|
||||
typing p99 33.2→26.7. Residual gaps remain: entrance/keyboard had 10/3 gaps over
|
||||
50 ms; typing had 30 over 25 ms (baseline 19). Keyboard p95 was essentially flat
|
||||
and typing p95 did not improve. These single runs establish removed transcript
|
||||
work and a safer entrance improvement, not universally smooth input or device FPS.
|
||||
The faster lazy option was rejected because strict reading/prepend anchors failed.
|
||||
|
||||
@@ -72,6 +72,116 @@ final class AgentPerformanceTests: XCTestCase {
|
||||
attachment.name = "agent-performance"; attachment.lifetime = .keepAlways; add(attachment)
|
||||
}
|
||||
|
||||
/// Exercises UIKit editing/focus and the real message reducer; no synthetic
|
||||
/// view rebuild loop or replacement composer/animation is used.
|
||||
func testInputAndEntranceResponsiveness() async throws {
|
||||
let room = SybilAgentRoom()
|
||||
try room.receive(.messages(AgentPage(conversationId: "performance",
|
||||
messages: (1...160).map(Self.message), cursor: "160", beforeCursor: nil, hasMore: false)), replay: true)
|
||||
room.flush()
|
||||
let host = UIHostingController(rootView: SybilAgentView(room: room, topInset: 64)
|
||||
.background(SybilTheme.background).environment(\.colorScheme, .dark))
|
||||
let scene = try XCTUnwrap(UIApplication.shared.connectedScenes.compactMap { $0 as? UIWindowScene }.first)
|
||||
let window = UIWindow(windowScene: scene)
|
||||
window.rootViewController = host
|
||||
window.makeKeyAndVisible()
|
||||
defer {
|
||||
host.view.endEditing(true)
|
||||
AgentRenderDiagnostics.enabled = false
|
||||
window.isHidden = true; window.rootViewController = nil
|
||||
room.stop()
|
||||
}
|
||||
host.view.frame = window.bounds
|
||||
host.view.layoutIfNeeded()
|
||||
try await Task.sleep(for: .seconds(2))
|
||||
let scroll = try XCTUnwrap(findScroll(host.view))
|
||||
let input = try XCTUnwrap(findInput(host.view))
|
||||
AgentRenderDiagnostics.enabled = true
|
||||
var reports: [[String: Any]] = []
|
||||
func report(_ sampler: AgentFrameSampler, phase: String) {
|
||||
var value = sampler.stop()
|
||||
value["phase"] = phase
|
||||
value["procedure"] = "mixed160-input-entrance-v1"
|
||||
value["operations"] = AgentRenderDiagnostics.counts
|
||||
value["contentHeight"] = scroll.contentSize.height
|
||||
value["viewportHeight"] = scroll.bounds.height
|
||||
reports.append(value)
|
||||
}
|
||||
|
||||
AgentRenderDiagnostics.counts = [:]
|
||||
let entrance = AgentFrameSampler(scroll: nil)
|
||||
entrance.start()
|
||||
for index in 161...172 {
|
||||
try room.receive(.messages(AgentPage(conversationId: "performance",
|
||||
messages: [Self.message(index)], cursor: String(index), beforeCursor: nil, hasMore: false)))
|
||||
// Use the production 100 ms publication batch and 240 ms entrance.
|
||||
try await Task.sleep(for: .milliseconds(550))
|
||||
assertAtBottom(scroll)
|
||||
}
|
||||
XCTAssertLessThanOrEqual(AgentRenderDiagnostics.counts["messageProjection", default: 0], 24,
|
||||
"Entrance frames must not repeatedly project the history")
|
||||
report(entrance, phase: "message-entrance")
|
||||
XCTAssertEqual(room.snapshot.room.messages.count, 172)
|
||||
assertAtBottom(scroll)
|
||||
capture(host, "input-entrance-latest")
|
||||
|
||||
let unfocusedHeight = scroll.bounds.height
|
||||
AgentRenderDiagnostics.counts = [:]
|
||||
let keyboard = AgentFrameSampler(scroll: nil)
|
||||
keyboard.start()
|
||||
for _ in 0..<6 {
|
||||
XCTAssertTrue(input.becomeFirstResponder())
|
||||
try await Task.sleep(for: .milliseconds(600))
|
||||
XCTAssertLessThan(scroll.bounds.height, unfocusedHeight - 100, "Software keyboard must actually resize the transcript")
|
||||
assertAtBottom(scroll)
|
||||
XCTAssertTrue(input.resignFirstResponder())
|
||||
try await Task.sleep(for: .milliseconds(600))
|
||||
}
|
||||
XCTAssertEqual(AgentRenderDiagnostics.counts["messageProjection", default: 0], 0,
|
||||
"Keyboard focus and geometry must not project the transcript")
|
||||
report(keyboard, phase: "keyboard-presentation")
|
||||
XCTAssertEqual(scroll.bounds.height, unfocusedHeight, accuracy: 1)
|
||||
XCTAssertTrue(input.becomeFirstResponder())
|
||||
try await Task.sleep(for: .milliseconds(700))
|
||||
let keyInput = try XCTUnwrap(input as? any UIKeyInput)
|
||||
AgentRenderDiagnostics.counts = [:]
|
||||
let typing = AgentFrameSampler(scroll: nil)
|
||||
typing.start()
|
||||
let text = "Typing into the actual composer should keep the long transcript responsive as this sentence wraps across several lines. "
|
||||
for character in text + text {
|
||||
keyInput.insertText(String(character))
|
||||
try await Task.sleep(for: .milliseconds(30))
|
||||
}
|
||||
XCTAssertEqual(AgentRenderDiagnostics.counts["messageProjection", default: 0], 0,
|
||||
"Draft edits and line wrapping must not project the transcript")
|
||||
assertAtBottom(scroll)
|
||||
report(typing, phase: "composer-typing")
|
||||
XCTAssertEqual(room.composer, text + text)
|
||||
capture(host, "input-multiline-keyboard")
|
||||
let data = try JSONSerialization.data(withJSONObject: reports, options: [.sortedKeys])
|
||||
print("AGENT_INPUT_PERFORMANCE_JSON " + String(decoding: data, as: UTF8.self))
|
||||
let attachment = XCTAttachment(data: data, uniformTypeIdentifier: "public.json")
|
||||
attachment.name = "agent-input-performance"; attachment.lifetime = .keepAlways; add(attachment)
|
||||
}
|
||||
|
||||
private func assertAtBottom(_ scroll: UIScrollView, file: StaticString = #filePath, line: UInt = #line) {
|
||||
XCTAssertEqual(scroll.contentOffset.y + scroll.bounds.height - scroll.adjustedContentInset.bottom,
|
||||
scroll.contentSize.height, accuracy: 25, file: file, line: line)
|
||||
}
|
||||
|
||||
private func findInput(_ view: UIView) -> UIView? {
|
||||
if view is UITextView || view is UITextField { return view }
|
||||
return view.subviews.lazy.compactMap { self.findInput($0) }.first
|
||||
}
|
||||
|
||||
private func capture(_ host: UIViewController, _ name: String) {
|
||||
let image = UIGraphicsImageRenderer(bounds: host.view.bounds).image { _ in
|
||||
host.view.drawHierarchy(in: host.view.bounds, afterScreenUpdates: true)
|
||||
}
|
||||
let attachment = XCTAttachment(image: image)
|
||||
attachment.name = name; attachment.lifetime = .keepAlways; add(attachment)
|
||||
}
|
||||
|
||||
private static func message(_ index: Int) -> Message {
|
||||
let own = index % 3 == 0
|
||||
let content = own
|
||||
|
||||
@@ -321,7 +321,7 @@ final class AgentUITests: XCTestCase {
|
||||
XCTAssertEqual(anchor.frame.minY, before, accuracy: 2)
|
||||
capture("agent-reading-anchor")
|
||||
for _ in 0..<25 {
|
||||
if app.buttons["Load earlier messages"].isHittable { break }
|
||||
if app.buttons["Load earlier messages"].exists && app.buttons["Load earlier messages"].isHittable { break }
|
||||
scroll.swipeDown(velocity: .fast)
|
||||
}
|
||||
XCTAssertTrue(app.buttons["Load earlier messages"].isHittable)
|
||||
|
||||
@@ -37,8 +37,8 @@ final class TranscriptFadeUITests: XCTestCase {
|
||||
}
|
||||
|
||||
private var transcript: XCUIElement {
|
||||
// The tablet sidebar is also a scroll view; select the detail by its content.
|
||||
app.scrollViews.containing(.staticText, identifier: "History message 160: local simulator fixture.").firstMatch
|
||||
// Identify the transcript independently of any particular message row.
|
||||
app.scrollViews["agent-transcript"]
|
||||
}
|
||||
|
||||
private func capture(_ name: String) {
|
||||
@@ -107,7 +107,7 @@ final class TranscriptFadeUITests: XCTestCase {
|
||||
capture("agent-fade-reading-position-and-unread")
|
||||
|
||||
for _ in 0..<30 {
|
||||
if app.buttons["Load earlier messages"].isHittable { break }
|
||||
if app.buttons["Load earlier messages"].exists && app.buttons["Load earlier messages"].isHittable { break }
|
||||
scroll.swipeDown(velocity: .fast)
|
||||
}
|
||||
let load = app.buttons["Load earlier messages"]
|
||||
|
||||
Reference in New Issue
Block a user