95 lines
5.2 KiB
Markdown
95 lines
5.2 KiB
Markdown
# Agent interaction benchmarks
|
||
|
||
Run the production Agent view in an app-hosted XCTest on the iPhone 16e simulator:
|
||
|
||
```sh
|
||
just generate
|
||
xcodebuild test -scheme SybilPresentation \
|
||
-destination 'platform=iOS Simulator,name=iPhone 16e,OS=latest' \
|
||
-derivedDataPath build/DerivedData -parallel-testing-enabled NO \
|
||
-only-testing:SybilPresentationTests/AgentPerformanceTests/testInputAndEntranceResponsiveness \
|
||
-resultBundlePath build/agent-input-performance.xcresult
|
||
```
|
||
|
||
`testInputAndEntranceResponsiveness` seeds 160 local mixed literal/Markdown
|
||
messages through the normal room reducer (100 initially displayed), then:
|
||
|
||
- Receives 12 more messages at 550 ms intervals through the production 100 ms
|
||
publication batch and 240 ms entrance animation, checking bottom following.
|
||
- Presents and dismisses the actual software keyboard six times, 600 ms per
|
||
transition. It asserts that the transcript viewport shrinks by over 100 points
|
||
and remains at the bottom; a hardware-keyboard-only run is invalid.
|
||
- Calls UIKit `insertText` on the actual focused composer every 30 ms, including
|
||
enough text to wrap and grow to its six-line limit. It verifies the draft,
|
||
bottom following, and that keyboard/draft changes don't reproject the history.
|
||
|
||
The real composer, Markdown rows, avatars and entrance animations remain enabled.
|
||
No server, production credentials, notification delivery or network images are
|
||
needed. The log prints `AGENT_INPUT_PERFORMANCE_JSON`; the result bundle retains
|
||
JSON and screenshots of the actual hosted view before and after typing.
|
||
Screenshots capture the hosting view, not the separate system keyboard window.
|
||
|
||
Use the same test source, simulator, build configuration and host load for
|
||
before/after measurements. Don't run other builds or profilers during sampling.
|
||
Functional assertions about observation boundaries should fail against the old
|
||
implementation; use the same sampling procedure when collecting a baseline.
|
||
|
||
## Measurement limits
|
||
|
||
Delivery-gap p50/p95/p99/max and counts over 25/50 ms measure **CADisplayLink
|
||
main-run-loop callback delivery**, not compositor FPS, GPU hitches, touch-to-key
|
||
latency or physical-device performance. The hosted view excludes the surrounding
|
||
app shell. UIKit input exercises editing/layout without synthesizing key taps.
|
||
The first keyboard presentation includes system warmup; residual system and
|
||
simulator scheduling gaps remain. Native gestures, read settling, header fade,
|
||
attachments and reading anchors are covered separately by the loopback UI tests.
|
||
|
||
`AgentRenderDiagnostics` counters are Debug-only and opt-in. They explain work;
|
||
a drop in counts alone does not establish improved responsiveness.
|
||
|
||
The shipped transcript retains its eager stack and exact row geometry. A lazy
|
||
experiment reduced measured work but failed the existing 2–3 point prepend
|
||
anchor checks. Retaining a pending anchor across geometry updates also failed;
|
||
an indefinite correction loop is not acceptable. The eager fallback keeps the
|
||
existing one-shot correction, isolates composer/read observation, and removes
|
||
row frame observers while following the bottom.
|
||
|
||
The experimental lazy content height changed from 43869 to 35870 points because
|
||
unmaterialized rows use estimates, not because content was shortened. Its
|
||
after-arrival screenshot was byte-identical to the eager baseline. Row text,
|
||
spacing, attachments and entrance growth were unchanged; the anchor failure
|
||
was sufficient to reject the lazy variant.
|
||
|
||
## Scrolling harness
|
||
|
||
`testThinkingScrollResponsiveness` retains the earlier three approximately
|
||
12-second phases: idle scroll, stationary thinking, and scrolling with changing
|
||
queue/worker/tool activity every 500 ms. Its display link sweeps 2400 points at
|
||
600 points/second. This programmatic sweep does not reproduce native drag phases;
|
||
use the UI tests for reading-anchor geometry and interactive keyboard dismissal.
|
||
|
||
The eager-stack reference in `b9a6f5d` has exact heights of 39181.333
|
||
points idle and 39223.333 while thinking, retained as regression assertions. That earlier scrolling fix reduced busy-scroll p95 from 88.8 to
|
||
19.4 ms; it did not address the input/entrance paths measured here. Physical
|
||
phone validation remains necessary for hardware performance claims.
|
||
|
||
## 2026-10-16 focused comparison
|
||
|
||
Single Debug simulator runs on the same iPhone 16e / iOS 26.5, starting at
|
||
`7510603` (which includes `b9a6f5d`). Values are callback-gap p95 milliseconds:
|
||
|
||
| Path | Before | Shipped eager isolation | Rejected lazy isolation |
|
||
| --- | ---: | ---: | ---: |
|
||
| 12 message entrances | 40.8 | 33.2 | 16.8 |
|
||
| Six keyboard cycles | 33.2 | 33.1 | 29.2 |
|
||
| Multiline composer typing | 24.8 | 25.2 | 21.3 |
|
||
|
||
Final transcript projections were 12/0/0 versus 220/24/10, with no row-frame
|
||
preference callbacks in these pinned-bottom phases. Eager content height stayed
|
||
43869.333 points. Entrance p99 improved 83.4→57.1 ms; keyboard p99 55.9→44.5;
|
||
typing p99 33.2→26.7. Residual gaps remain: entrance/keyboard had 10/3 gaps over
|
||
50 ms; typing had 30 over 25 ms (baseline 19). Keyboard p95 was essentially flat
|
||
and typing p95 did not improve. These single runs establish removed transcript
|
||
work and a safer entrance improvement, not universally smooth input or device FPS.
|
||
The faster lazy option was rejected because strict reading/prepend anchors failed.
|