Support durable assistant image attachments in Agent clients

This commit is contained in:
2026-10-20 14:14:09 -07:00
parent 7b48201e28
commit 489c347976
15 changed files with 227 additions and 40 deletions
+44 -1
View File
@@ -708,11 +708,54 @@ Global `SYBIL_AGENT_ATTACHMENT_QUOTA_BYTES` defaults to 1 GiB. Concurrent reserv
Release requires native `SYBIL_AGENT_AUTH_MODE=token` with nonempty `SYBIL_AGENT_API_KEY`, plus backend `ADMIN_TOKEN` and matching server-side `SYBIL_AGENT_API_KEY` / `SYBIL_AGENT_BASE_URL`. Native open mode cannot advertise/store/download attachments. A runtime containing attachment records/files refuses open startup; raw/model APIs and sockets are protected in token mode. An open Sybil proxy also refuses access to an attachment-capable native runtime, rather than exposing its history through a private server credential. Explicit text-only open development remains supported with an open/legacy runtime lacking attachment capability. Native keys never enter browser/iOS payloads. No insecure development opt-in is enabled.
The runtime source is built into its container: rebuild it with pinned busboy/sharp dependencies for the target Node/Linux platform before release. The deployed open auth mode must be changed separately; no deployed config or mount is modified by this implementation. Backend and web must be upgraded with runtime capability/auth configured before clients advertise uploads.
The runtime source is built into its container: rebuild it with pinned busboy/sharp dependencies for the target Node/Linux platform before release. Deployment must use native token auth and matching backend credentials before clients advertise attachments; no additional public mount is needed.
Browser drafts retain upload IDs and completed descriptors across pane changes and reload. Incomplete uploads require reselection of the original filename/type/size and, once network delivery has begun, matching SHA-256 bytes; retries reuse uploadId. Sending persists the attachment-bearing outbox before clearing the draft. Message confirmation requires the same ordered attachment IDs in receipts and canonical echoes. Mismatched echoes must retain local attachments and delivery errors; they are not acknowledgement of a file submission.
### Assistant image output (v1)
The independent optional capability `capabilities.assistantImages: {"version":1}`
advertises the runtime's root-only `output_image` tool. It does **not** enable input
upload controls; those still require `capabilities.attachments.version == 1`.
Authenticated downloads accept either supported capability. Open proxies refuse
history and socket access if either private attachment capability is advertised.
Capability absence does not hide previously stored assistant image messages.
The agent invokes `output_image({path, uploadId, filename?, caption?})` with a stable
UUID retry identity and a local persistent raster exported beneath the workspace's
`artifacts/` or `downloads/` directory. It copies validated bytes into the existing
attachment store and publishes one durable assistant message. This is an internal
agent tool, with no browser/iOS publishing endpoint or client-supplied role.
Workers cannot publish through it. No URL fetch, remote image proxy, file execution
or implicit forwarding of native tool image blocks is performed. Generation is
available only when a separate generation provider/tool is actually installed.
Output uses the same original-byte/hash/quota limits (PNG/JPEG/GIF/WebP, 6 MiB,
50 million pixels, 16,384 pixels per axis), authenticated content route and backup
policy as uploads. Sources must be confined regular files without symlink/hardlink
substitution or private path segments. Upload IDs are bound to output provenance;
conflicting arguments fail and identical retries retain their message/attachment IDs.
Retries reuse the stored snapshot even if the original source moves or changes;
publishing changed bytes requires a new upload UUID.
Public output rows have `id: "outbound:<entryId>"`, `role: "assistant"`,
`metadata.kind: "message"`, `metadata.status: "complete"`, and
`metadata.agentAttachments` containing image descriptors. `content` is the optional
caption, otherwise an empty string; attachment-only rows remain visible. Descriptors
contain only `id`, `kind`, safe `filename`, `mimeType`, `sizeBytes` and `sha256`.
They contain no workspace/source path, URL, credential, or image bytes. The public
tool row redacts output tool arguments/results; internal task/submission provenance
stays in the existing durable outbox.
Web/iOS automatically render assistant rasters in fixed preview frames, retrieving
only authenticated ID-based originals, checking size and SHA-256, rejecting
redirects, and offering retry/download. Web limits downloads to two at a time and
uses revocable blob URLs. Clients discard assistant descriptor paths and unsupported
types. Agent Markdown cannot load images; external links remain explicit navigation.
Image-bearing push alerts use **You have a new message.**, including when a caption
exists, and never include attachment names, paths, bytes or URLs.
## Agent push notifications (v1)
All push/config/presence routes require a configured `ADMIN_TOKEN` and bearer authentication; open development mode returns 503. Missing/wrong bearer returns 401/403. Bodies are limited to 16 KiB, responses use `Cache-Control: no-store`, invalid bodies return 400, and unavailable configuration returns 503. Do not put credentials, device tokens or subscription endpoints into URLs. Ownership is one **shared authenticated personal principal**, derived on the server from the configured credential generation. Anyone possessing that shared token has authority over all its installations; UUIDs identify installations and do not create per-human isolation. No `ownerId` is accepted. Changing `ADMIN_TOKEN` invalidates all prior push opt-ins. Protect the SQLite database/backups because endpoint credentials are stored there.
+14
View File
@@ -351,6 +351,20 @@ for unloaded-history count semantics and rendering limits.
### Agent attachment history, queued activity and replay
Assistant image output uses the same `messages` frames and durable history rows,
with `role: "assistant"`, `id: "outbound:<entryId>"`, `metadata.kind: "message"`,
`metadata.status: "complete"`, and `metadata.agentAttachments`. Each image descriptor
contains `id`, `kind: "image"`, `filename`, `mimeType`, `sizeBytes` and `sha256` only;
there is no source/workspace path or URL. An omitted caption yields empty `content`;
clients must retain this image-only row. Merge/replay by the unchanged message ID,
render authenticated image bytes inline in a fixed frame, and retain loaded history
even if the optional `capabilities.assistantImages.version: 1` is absent later.
That capability does not authorize input uploads. Native tool image blocks are not
automatically forwarded; the root agent explicitly calls `output_image`. Public
output tool rows redact arguments/results, and Agent Markdown does not fetch images.
See [assistant image output](./rest.md#assistant-image-output-v1) for publishing,
retry, auth and generic push-preview semantics. No new socket event is introduced.
Uploads and attached sends use REST, as specified in [rest.md](./rest.md#agent-attachments-v1). WebSocket authentication, event types, stable message IDs, cursor semantics and raw tool rows are unchanged. Capability discovery is optional `GET /v1/agent` → `capabilities.attachments.version: 1`; absence disables new attachment sends. Upgrade/authenticate the native runtime and backend before advertising it.
Both `messages.messages[]` and `activity.activity.queuedMessages[]` carry attachment descriptors under **`metadata.agentAttachments`**, distinct from ordinary chat's inline `metadata.attachments`: