108 lines
6.0 KiB
Markdown
108 lines
6.0 KiB
Markdown
# Agent interaction benchmarks
|
|
|
|
Run the production Agent view in an app-hosted XCTest on the iPhone 16e simulator:
|
|
|
|
```sh
|
|
just generate
|
|
xcodebuild test -scheme SybilPresentation \
|
|
-destination 'platform=iOS Simulator,name=iPhone 16e,OS=latest' \
|
|
-derivedDataPath build/DerivedData -parallel-testing-enabled NO \
|
|
-only-testing:SybilPresentationTests/AgentPerformanceTests/testInputAndEntranceResponsiveness \
|
|
-resultBundlePath build/agent-input-performance.xcresult
|
|
```
|
|
|
|
`testInputAndEntranceResponsiveness` seeds 160 local mixed literal/Markdown
|
|
messages through the normal room reducer (100 initially displayed), then:
|
|
|
|
- Receives 12 more messages at 550 ms intervals through the production 100 ms
|
|
publication batch and 240 ms entrance animation, checking bottom following.
|
|
- Presents and dismisses the actual software keyboard six times, 600 ms per
|
|
transition. It asserts that the transcript viewport shrinks by over 100 points
|
|
and remains at the bottom; a hardware-keyboard-only run is invalid.
|
|
- Calls UIKit `insertText` on the actual focused composer every 30 ms, including
|
|
enough text to wrap and grow to its six-line limit. It verifies the draft,
|
|
bottom following, and that keyboard/draft changes don't reproject the history.
|
|
|
|
The real composer, Markdown rows, avatars and entrance animations remain enabled.
|
|
No server, production credentials, notification delivery or network images are
|
|
needed. The log prints `AGENT_INPUT_PERFORMANCE_JSON`; the result bundle retains
|
|
JSON and screenshots of the actual hosted view before and after typing.
|
|
Screenshots capture the hosting view, not the separate system keyboard window.
|
|
|
|
Use the same test source, simulator, build configuration and host load for
|
|
before/after measurements. Don't run other builds or profilers during sampling.
|
|
Functional assertions about observation boundaries should fail against the old
|
|
implementation; use the same sampling procedure when collecting a baseline.
|
|
|
|
## Measurement limits
|
|
|
|
Delivery-gap p50/p95/p99/max and counts over 25/50 ms measure **CADisplayLink
|
|
main-run-loop callback delivery**, not compositor FPS, GPU hitches, touch-to-key
|
|
latency or physical-device performance. The hosted view excludes the surrounding
|
|
app shell. UIKit input exercises editing/layout without synthesizing key taps.
|
|
The first keyboard presentation includes system warmup; residual system and
|
|
simulator scheduling gaps remain. Native gestures, read settling, header fade,
|
|
attachments and reading anchors are covered separately by the loopback UI tests.
|
|
|
|
`AgentRenderDiagnostics` counters are Debug-only and opt-in. They explain work;
|
|
a drop in counts alone does not establish improved responsiveness.
|
|
|
|
The transcript now uses `LazyVStack` with the observation boundaries introduced
|
|
in `bbaa5e8`. Loading older history may move the viewport; this is the only
|
|
waived position guarantee. Native regressions still assert reading anchors for
|
|
incoming messages, bottom following, keyboard behavior, read settling, fade,
|
|
and attachments. There is no saved prepend anchor or convergence loop.
|
|
|
|
Lazy content height estimates change as offscreen rows materialize. The earlier
|
|
43869→35870 point difference was not shortened content: after-arrival screenshots
|
|
were byte-identical. Compare visible geometry and native behavior, not estimated
|
|
whole-history height, when assessing lazy performance.
|
|
|
|
## Scrolling harness
|
|
|
|
`testThinkingScrollResponsiveness` retains the earlier three approximately
|
|
12-second phases: idle scroll, stationary thinking, and scrolling with changing
|
|
queue/worker/tool activity every 500 ms. Its display link sweeps 2400 points at
|
|
600 points/second. This programmatic sweep does not reproduce native drag phases;
|
|
use the UI tests for reading-anchor geometry and interactive keyboard dismissal.
|
|
|
|
The eager-stack reference in `b9a6f5d` has exact heights of 39181.333
|
|
points idle and 39223.333 while thinking. That reference harness retains its
|
|
eager-only total-height assertions; run it at `bbaa5e8` for the eager comparison.
|
|
Those totals are not valid assertions about lazy estimated content size. Use the
|
|
input harness and native gesture tests above for the current lazy implementation.
|
|
That earlier scrolling fix reduced busy-scroll p95 from 88.8 to 19.4 ms; it did
|
|
not address the input/entrance paths measured here. Physical phone validation
|
|
remains necessary for hardware performance claims.
|
|
|
|
## 2026-10-16 comparison before the prepend waiver
|
|
|
|
Single Debug simulator runs on the same iPhone 16e / iOS 26.5, starting at
|
|
`7510603` (which includes `b9a6f5d`). Values are callback-gap p95 milliseconds:
|
|
|
|
| Path | Before | Eager isolation (`bbaa5e8`) | Earlier lazy experiment |
|
|
| --- | ---: | ---: | ---: |
|
|
| 12 message entrances | 40.8 | 33.2 | 16.8 |
|
|
| Six keyboard cycles | 33.2 | 33.1 | 29.2 |
|
|
| Multiline composer typing | 24.8 | 25.2 | 21.3 |
|
|
|
|
Eager transcript projections were 12/0/0 versus 220/24/10, with no row-frame
|
|
preference callbacks in these pinned-bottom phases. Eager content height stayed
|
|
43869.333 points. Entrance p99 improved 83.4→57.1 ms; keyboard p99 55.9→44.5;
|
|
typing p99 33.2→26.7. Residual gaps remain: entrance/keyboard had 10/3 gaps over
|
|
50 ms; typing had 30 over 25 ms (baseline 19). Keyboard p95 was essentially flat
|
|
and typing p95 did not improve. These single runs establish removed transcript
|
|
work and a safer entrance improvement, not universally smooth input or device FPS.
|
|
The lazy option was initially rejected because strict prepend anchors failed.
|
|
The subsequent release accepts that prepend tradeoff; all other guarantees remain.
|
|
|
|
## Lazy release after the prepend waiver
|
|
|
|
The same input harness passes unchanged: p95 entrance/keyboard/typing gaps are
|
|
16.9/28.1/20.9 ms, versus the eager run's 33.2/33.1/25.2 ms. Projections stay
|
|
12/0/0; pinned row-frame preference callbacks stay zero. The after-arrival PNG
|
|
is byte-identical to the eager baseline. Residual gaps remain (one keyboard gap
|
|
over 50 ms and one typing gap over 25 ms); these are single simulator samples. Five
|
|
focused native regressions pass with only the two prepend-offset assertions
|
|
replaced by checks that the remaining history loaded.
|