diff --git a/docs/api/rest.md b/docs/api/rest.md index 5097897..e9c0f81 100644 --- a/docs/api/rest.md +++ b/docs/api/rest.md @@ -708,11 +708,54 @@ Global `SYBIL_AGENT_ATTACHMENT_QUOTA_BYTES` defaults to 1 GiB. Concurrent reserv Release requires native `SYBIL_AGENT_AUTH_MODE=token` with nonempty `SYBIL_AGENT_API_KEY`, plus backend `ADMIN_TOKEN` and matching server-side `SYBIL_AGENT_API_KEY` / `SYBIL_AGENT_BASE_URL`. Native open mode cannot advertise/store/download attachments. A runtime containing attachment records/files refuses open startup; raw/model APIs and sockets are protected in token mode. An open Sybil proxy also refuses access to an attachment-capable native runtime, rather than exposing its history through a private server credential. Explicit text-only open development remains supported with an open/legacy runtime lacking attachment capability. Native keys never enter browser/iOS payloads. No insecure development opt-in is enabled. -The runtime source is built into its container: rebuild it with pinned busboy/sharp dependencies for the target Node/Linux platform before release. The deployed open auth mode must be changed separately; no deployed config or mount is modified by this implementation. Backend and web must be upgraded with runtime capability/auth configured before clients advertise uploads. +The runtime source is built into its container: rebuild it with pinned busboy/sharp dependencies for the target Node/Linux platform before release. Deployment must use native token auth and matching backend credentials before clients advertise attachments; no additional public mount is needed. Browser drafts retain upload IDs and completed descriptors across pane changes and reload. Incomplete uploads require reselection of the original filename/type/size and, once network delivery has begun, matching SHA-256 bytes; retries reuse uploadId. Sending persists the attachment-bearing outbox before clearing the draft. Message confirmation requires the same ordered attachment IDs in receipts and canonical echoes. Mismatched echoes must retain local attachments and delivery errors; they are not acknowledgement of a file submission. +### Assistant image output (v1) + +The independent optional capability `capabilities.assistantImages: {"version":1}` +advertises the runtime's root-only `output_image` tool. It does **not** enable input +upload controls; those still require `capabilities.attachments.version == 1`. +Authenticated downloads accept either supported capability. Open proxies refuse +history and socket access if either private attachment capability is advertised. +Capability absence does not hide previously stored assistant image messages. + +The agent invokes `output_image({path, uploadId, filename?, caption?})` with a stable +UUID retry identity and a local persistent raster exported beneath the workspace's +`artifacts/` or `downloads/` directory. It copies validated bytes into the existing +attachment store and publishes one durable assistant message. This is an internal +agent tool, with no browser/iOS publishing endpoint or client-supplied role. +Workers cannot publish through it. No URL fetch, remote image proxy, file execution +or implicit forwarding of native tool image blocks is performed. Generation is +available only when a separate generation provider/tool is actually installed. + +Output uses the same original-byte/hash/quota limits (PNG/JPEG/GIF/WebP, 6 MiB, +50 million pixels, 16,384 pixels per axis), authenticated content route and backup +policy as uploads. Sources must be confined regular files without symlink/hardlink +substitution or private path segments. Upload IDs are bound to output provenance; +conflicting arguments fail and identical retries retain their message/attachment IDs. +Retries reuse the stored snapshot even if the original source moves or changes; +publishing changed bytes requires a new upload UUID. + +Public output rows have `id: "outbound:"`, `role: "assistant"`, +`metadata.kind: "message"`, `metadata.status: "complete"`, and +`metadata.agentAttachments` containing image descriptors. `content` is the optional +caption, otherwise an empty string; attachment-only rows remain visible. Descriptors +contain only `id`, `kind`, safe `filename`, `mimeType`, `sizeBytes` and `sha256`. +They contain no workspace/source path, URL, credential, or image bytes. The public +tool row redacts output tool arguments/results; internal task/submission provenance +stays in the existing durable outbox. + +Web/iOS automatically render assistant rasters in fixed preview frames, retrieving +only authenticated ID-based originals, checking size and SHA-256, rejecting +redirects, and offering retry/download. Web limits downloads to two at a time and +uses revocable blob URLs. Clients discard assistant descriptor paths and unsupported +types. Agent Markdown cannot load images; external links remain explicit navigation. +Image-bearing push alerts use **You have a new message.**, including when a caption +exists, and never include attachment names, paths, bytes or URLs. + ## Agent push notifications (v1) All push/config/presence routes require a configured `ADMIN_TOKEN` and bearer authentication; open development mode returns 503. Missing/wrong bearer returns 401/403. Bodies are limited to 16 KiB, responses use `Cache-Control: no-store`, invalid bodies return 400, and unavailable configuration returns 503. Do not put credentials, device tokens or subscription endpoints into URLs. Ownership is one **shared authenticated personal principal**, derived on the server from the configured credential generation. Anyone possessing that shared token has authority over all its installations; UUIDs identify installations and do not create per-human isolation. No `ownerId` is accepted. Changing `ADMIN_TOKEN` invalidates all prior push opt-ins. Protect the SQLite database/backups because endpoint credentials are stored there. diff --git a/docs/api/streaming-chat.md b/docs/api/streaming-chat.md index 4ba850e..58e35d8 100644 --- a/docs/api/streaming-chat.md +++ b/docs/api/streaming-chat.md @@ -351,6 +351,20 @@ for unloaded-history count semantics and rendering limits. ### Agent attachment history, queued activity and replay +Assistant image output uses the same `messages` frames and durable history rows, +with `role: "assistant"`, `id: "outbound:"`, `metadata.kind: "message"`, +`metadata.status: "complete"`, and `metadata.agentAttachments`. Each image descriptor +contains `id`, `kind: "image"`, `filename`, `mimeType`, `sizeBytes` and `sha256` only; +there is no source/workspace path or URL. An omitted caption yields empty `content`; +clients must retain this image-only row. Merge/replay by the unchanged message ID, +render authenticated image bytes inline in a fixed frame, and retain loaded history +even if the optional `capabilities.assistantImages.version: 1` is absent later. +That capability does not authorize input uploads. Native tool image blocks are not +automatically forwarded; the root agent explicitly calls `output_image`. Public +output tool rows redact arguments/results, and Agent Markdown does not fetch images. +See [assistant image output](./rest.md#assistant-image-output-v1) for publishing, +retry, auth and generic push-preview semantics. No new socket event is introduced. + Uploads and attached sends use REST, as specified in [rest.md](./rest.md#agent-attachments-v1). WebSocket authentication, event types, stable message IDs, cursor semantics and raw tool rows are unchanged. Capability discovery is optional `GET /v1/agent` → `capabilities.attachments.version: 1`; absence disables new attachment sends. Upgrade/authenticate the native runtime and backend before advertising it. Both `messages.messages[]` and `activity.activity.queuedMessages[]` carry attachment descriptors under **`metadata.agentAttachments`**, distinct from ordinary chat's inline `metadata.attachments`: diff --git a/server/src/agent-attachments.ts b/server/src/agent-attachments.ts index 11c5f1c..c32f658 100644 --- a/server/src/agent-attachments.ts +++ b/server/src/agent-attachments.ts @@ -18,11 +18,12 @@ export type AgentProxyJson = (path: string, method?: string, body?: unknown, sig function apiError(statusCode: number, code: string, message: string) { return Object.assign(new Error(message), { statusCode, code }); } -export async function requireAgentAttachments(proxyJson: AgentProxyJson, signal?: AbortSignal) { +export async function requireAgentAttachments(proxyJson: AgentProxyJson, signal?: AbortSignal, download = false) { if (!env.ADMIN_TOKEN) throw apiError(503, 'attachments_unavailable', 'Attachments require authenticated Sybil access.'); const descriptor = await proxyJson('/api/conversations/root', 'GET', undefined, signal); - const value = descriptor.data as { capabilities?: { attachments?: { version?: number } } } | null; - if (descriptor.status !== 200 || value?.capabilities?.attachments?.version !== 1) + const value = descriptor.data as { capabilities?: { attachments?: { version?: number }; assistantImages?: { version?: number } } } | null; + if (descriptor.status !== 200 || !(value?.capabilities?.attachments?.version === 1 + || (download && value?.capabilities?.assistantImages?.version === 1))) throw apiError(503, 'attachments_unavailable', 'The Agent does not currently support authenticated attachments.'); } @@ -52,7 +53,7 @@ export async function registerAgentAttachmentRoutes(app: FastifyInstance, option if (req.raw.aborted || reply.raw.destroyed) { release(); controller.signal.throwIfAborted(); } if (uploading && Number(req.headers['content-length'] || 0) > AGENT_UPLOAD_LIMIT) throw apiError(413, 'attachment_limit_exceeded', 'Upload request exceeds its size limit.'); - await requireAgentAttachments(options.proxyJson, controller.signal); + await requireAgentAttachments(options.proxyJson, controller.signal, !uploading); controller.signal.throwIfAborted(); }); routes.addHook('onResponse', async req => { transfers.get(req)?.release(); }); diff --git a/server/src/agent-push-preview.ts b/server/src/agent-push-preview.ts index f72c1cf..6764f2c 100644 --- a/server/src/agent-push-preview.ts +++ b/server/src/agent-push-preview.ts @@ -14,7 +14,8 @@ export type MessageText = (message: PublicMessage) => Promise; const Page = z.object({ conversationId: z.string(), cursor: Cursor, beforeCursor: Cursor, messages: z.array(z.unknown()).max(1000) }); const Message = z.object({ id: z.string(), role: z.literal('assistant'), content: z.string(), - metadata: z.object({ kind: z.literal('message'), status: z.literal('complete'), entryId: z.number().int().safe() }) }); + metadata: z.object({ kind: z.literal('message'), status: z.literal('complete'), entryId: z.number().int().safe(), + agentAttachments: z.unknown().optional() }) }); /** Resolve only an already eligible candidate from one immutable root public entry. */ export function createMessageText(upstream: { baseUrl: string; apiKey?: string }): MessageText { @@ -47,7 +48,10 @@ export function createMessageText(upstream: { baseUrl: string; apiKey?: string } const matches = page.messages.filter(value => value && typeof value === 'object' && 'id' in value && value.id === candidate.messageId); const message = matches.length === 1 ? Message.safeParse(matches[0]) : undefined; if (!message?.success || message.data.metadata.entryId !== Number(candidate.cursor)) throw new Error('Public Agent message unavailable'); - // Never inspect metadata, tool results, activity, reasoning blocks or linked resources for text. + // Attachment messages use the generic alert. Never expose captions, filenames, + // paths, image data or linked resources in an image notification. + if (message.data.metadata.agentAttachments !== undefined) return ''; + // Never inspect tool results, activity, reasoning blocks or linked resources for text. return message.data.content; }; } diff --git a/server/src/agent.ts b/server/src/agent.ts index 56b8a92..0594069 100644 --- a/server/src/agent.ts +++ b/server/src/agent.ts @@ -43,8 +43,8 @@ export async function registerAgentRoutes(app: FastifyInstance, options: AgentOp const response = await fetch(new URL('/api/conversations/root', baseUrl), { headers: { 'X-Sybil-Client': 'backend-proxy-v1', Authorization: `Bearer ${apiKey}` }, signal: signal ? AbortSignal.any([signal, AbortSignal.timeout(15_000)]) : AbortSignal.timeout(15_000), redirect: 'error', }); - const descriptor = await response.json() as { capabilities?: { attachments?: unknown } }; - if (!response.ok || descriptor?.capabilities?.attachments) throw new Error('Private runtime'); + const descriptor = await response.json() as { capabilities?: { attachments?: unknown; assistantImages?: unknown } }; + if (!response.ok || descriptor?.capabilities?.attachments || descriptor?.capabilities?.assistantImages) throw new Error('Private runtime'); } catch { throw Object.assign(new Error('Authenticated Sybil access is required for this Agent runtime.'), { statusCode: 503, code: 'attachments_unavailable' }); } }; const proxyJson = async (path: string, method = 'GET', body?: unknown, signal?: AbortSignal) => { @@ -53,6 +53,7 @@ export async function registerAgentRoutes(app: FastifyInstance, options: AgentOp let response: Response; try { response = await fetch(new URL(path, baseUrl), { method, + redirect: 'error', headers: { 'X-Sybil-Client': 'backend-proxy-v1', Authorization: `Bearer ${apiKey}`, ...(body === undefined ? {} : { 'Content-Type': 'application/json' }) }, ...(body === undefined ? {} : { body: JSON.stringify(body) }), signal: signal ? AbortSignal.any([signal, AbortSignal.timeout(15_000)]) : AbortSignal.timeout(15_000), }); diff --git a/server/tests/agent-attachments.test.ts b/server/tests/agent-attachments.test.ts index 7345835..6a75f98 100644 --- a/server/tests/agent-attachments.test.ts +++ b/server/tests/agent-attachments.test.ts @@ -20,14 +20,14 @@ const [{ registerAgentRoutes }, { env }] = await Promise.all([import('../src/age const auth = { authorization: `Bearer ${process.env.ADMIN_TOKEN}` }, key = 'runtime-private-test-key'; const file = { id:'att_12345678-1234-4234-8234-123456789abc',kind:'text',filename:'résumé.txt',mimeType:'text/plain',sizeBytes:4,sha256:'a'.repeat(64),workspacePath:'/data/workspace/uploads/att_12345678-1234-4234-8234-123456789abc/résumé.txt' }; const bodies: any[] = []; -let capable=true, badReceipt=false, errorStatus=0, uploadCalls=0; +let capable=true, assistantImages=false, badReceipt=false, errorStatus=0, uploadCalls=0; let holdDescriptor=false, holdDownload=false;const held:ServerResponse[]=[], closed:ServerResponse[]=[]; const source=createServer(async(req,res)=>{ assert.equal(req.headers.authorization,`Bearer ${key}`); const chunks: Buffer[]=[]; try { for await(const chunk of req) chunks.push(chunk); } catch { return; } const bytes=Buffer.concat(chunks);res.setHeader('Content-Type','application/json'); - if(req.url==='/api/conversations/root'){if(holdDescriptor){held.push(res);res.once('close',()=>closed.push(res));return;}res.end(JSON.stringify({id:'1',...(capable?{capabilities:{attachments:{version:1}}}:{})}));return;} + if(req.url==='/api/conversations/root'){if(holdDescriptor){held.push(res);res.once('close',()=>closed.push(res));return;}res.end(JSON.stringify({id:'1',capabilities:{...(capable?{attachments:{version:1}}:{}),...(assistantImages?{assistantImages:{version:1}}:{})}}));return;} if(errorStatus){res.statusCode=errorStatus;res.end(JSON.stringify({error:true,message:'Storage quota reached',statusCode:errorStatus,code:'attachment_quota_exceeded',remainingBytes:12,quotaBytes:1024,requiredBytes:99}));return;} if(req.url?.endsWith('/attachments')){uploadCalls++;assert.match(req.headers['content-type']!,/^multipart\/form-data; boundary=/);assert.ok(bytes.includes(Buffer.from('résumé.txt')));bodies.push(bytes.toString());res.statusCode=201;res.end(JSON.stringify({attachment:file}));return;} if(req.url?.endsWith('/content')){if(holdDownload){held.push(res);res.once('close',()=>closed.push(res));return;}res.writeHead(200,{'Content-Type':'text/plain; charset=utf-8','Content-Length':'4','Content-Disposition':"attachment; filename=notes.txt; filename*=UTF-8''notes.txt"});res.end('data');return;} @@ -57,6 +57,22 @@ test('multipart forwards Unicode filename/uploadId with private auth; quota deta assert.equal((await app.inject({url:'/v1/agent/attachments/not-an-id/content',headers:auth})).statusCode,404); }); +test('assistant-only capability allows authenticated retrieval without granting input uploads or open history',async()=>{ + capable=false;assistantImages=true; + const saved=env.ADMIN_TOKEN; + try { + const descriptor=(await app.inject({url:'/v1/agent',headers:auth})).json(); + assert.deepEqual(descriptor.capabilities,{assistantImages:{version:1}}); + assert.equal((await fetch(base+`/v1/agent/attachments/${file.id}/content`,{headers:auth})).status,200); + assert.equal((await fetch(base+`/v1/agent/attachments/${file.id}/content`)).status,401); + assert.equal((await fetch(base+'/v1/agent/attachments',{method:'POST',headers:auth,body:form()})).status,503); + assert.equal((await app.inject({method:'POST',url:'/v1/agent/messages',headers:auth,payload:{content:'',requestId:'assistant-only',attachments:[{id:file.id}]}})).statusCode,503); + assert.equal((await app.inject({method:'POST',url:'/v1/agent/output_image',headers:auth,payload:{path:'https://never-fetch.invalid/image'}})).statusCode,404); + env.ADMIN_TOKEN=undefined; + for(const url of ['/v1/agent','/v1/agent/messages',`/v1/agent/attachments/${file.id}/content`]) assert.equal((await app.inject({url})).statusCode,503); + } finally {env.ADMIN_TOKEN=saved;capable=true;assistantImages=false;} +}); + test('ID-only sends require matching receipts; history retains attachment metadata and raw tools',async()=>{ const payload={content:'',requestId:'send',attachments:[{id:file.id}]}; assert.equal((await app.inject({method:'POST',url:'/v1/agent/messages',headers:auth,payload})).statusCode,202);assert.deepEqual(bodies.at(-1),{...payload,channel:'sybil',whenBusy:'followUp',wait:false}); diff --git a/server/tests/agent-push-preview.test.ts b/server/tests/agent-push-preview.test.ts index 2457366..62d4d06 100644 --- a/server/tests/agent-push-preview.test.ts +++ b/server/tests/agent-push-preview.test.ts @@ -41,6 +41,11 @@ test('preview lookup uses only the exact eligible completed root assistant conte ]) { page = invalid; await assert.rejects(load(candidate)); } page = { ...original, messages: [{ ...message, content: '' }] }; assert.equal(await load(candidate), '', 'empty content never borrows metadata, tool output or reasoning'); + page = { ...original, messages: [{ ...message, content: 'Caption stays inside authenticated chat.', metadata: { ...message.metadata, + agentAttachments: [{ filename: 'private-name.png', workspacePath: '/private/source.png', url: 'https://never-fetch.invalid/image' }] } }] }; + const imageText = await load(candidate); + assert.equal(imageText, '', 'image captions and metadata use only a generic notification'); + assert.deepEqual(JSON.parse(previewPayload(imageText, body => ({ body }), 4096).toString()), { body: 'You have a new message.' }); const before = requests.length; for (const messageId of ['tool:1:11', 'outbound:12', 'https://never-fetch.invalid/', 'entry:11:0/../../state']) { await assert.rejects(load({ ...candidate, messageId })); diff --git a/web/src/components/agent/agent-attachment.tsx b/web/src/components/agent/agent-attachment.tsx index 0f5b888..8533c47 100644 --- a/web/src/components/agent/agent-attachment.tsx +++ b/web/src/components/agent/agent-attachment.tsx @@ -1,19 +1,20 @@ -import { useEffect, useRef, useState } from "preact/hooks"; +import { useCallback, useEffect, useRef, useState } from "preact/hooks"; import { Copy, Download, FileText, Image, LoaderCircle } from "lucide-preact"; import { attachmentSize, type AgentAttachment } from "@/lib/agent-attachments"; -type Props = { file: AgentAttachment; download(file: AgentAttachment, signal?: AbortSignal): Promise }; +type Props = { file: AgentAttachment; inline?: boolean; download(file: AgentAttachment, signal?: AbortSignal): Promise }; /** Authenticated bytes only; no native credential, remote href, or public file serving. */ -export function AgentAttachmentCard({ file, download }: Props) { +export function AgentAttachmentCard({ file, download, inline = false }: Props) { const [preview, setPreview] = useState(); const [error, setError] = useState(); const [busy, setBusy] = useState(false); const [copied, setCopied] = useState(false); const controller = useRef(); - useEffect(() => () => { controller.current?.abort(); }, []); + const transferring = useRef(false); useEffect(() => () => { if (preview) URL.revokeObjectURL(preview); }, [preview]); - const retrieve = async (show: boolean) => { - if (busy) return; + const retrieve = useCallback(async (show: boolean) => { + if (transferring.current) return; + transferring.current = true; const request = new AbortController(); controller.current = request; setBusy(true); setError(undefined); try { const blob = await download(file, request.signal); if (request.signal.aborted) return; @@ -24,14 +25,22 @@ export function AgentAttachmentCard({ file, download }: Props) { setTimeout(() => URL.revokeObjectURL(url), 1000); } } catch (failure) { if (!request.signal.aborted) setError(failure instanceof Error ? failure.message : "Download failed."); } - finally { if (!request.signal.aborted) setBusy(false); } - }; + finally { if (controller.current === request) { transferring.current = false; if (!request.signal.aborted) setBusy(false); } } + }, [download, file.id, file.sha256, file.mimeType, file.kind, file.sizeBytes, file.filename]); + useEffect(() => { + setPreview(undefined); setError(undefined); setBusy(false); transferring.current = false; + if (inline && file.kind === "image") void retrieve(true); + return () => { controller.current?.abort(); }; + }, [inline, retrieve]); return
{file.filename}{attachmentSize(file.sizeBytes)}
- {file.workspacePath ?
{file.workspacePath}
: null} - {preview ? {file.filename} : null} + {!inline && file.workspacePath ?
{file.workspacePath}
: null} + {inline ?
+ {preview ? {file.filename} { setPreview(undefined); setError("Image could not be displayed."); }} /> + : {error ? "Image unavailable" : "Loading image…"}} +
: preview ? {file.filename} : null}
- {file.kind === "image" ? : null} + {file.kind === "image" && (!inline || error) ? : null}
{error ? {error} Retry using the buttons above. : null} diff --git a/web/src/components/agent/agent-conversation.css b/web/src/components/agent/agent-conversation.css index 98d66ef..d3f6fe9 100644 --- a/web/src/components/agent/agent-conversation.css +++ b/web/src/components/agent/agent-conversation.css @@ -476,4 +476,6 @@ .agent-attachment__actions button { display: flex; align-items: center; gap: 5px; color: #ddd6fe; } .agent-attachment__actions button:disabled { opacity: .5; } .agent-attachment__preview { display: block; max-height: 260px; max-width: 100%; margin-top: 8px; border-radius: 6px; } +.agent-attachment__image-frame { display: flex; align-items: center; justify-content: center; height: 260px; width: min(480px, 100%); margin-top: 8px; } +.agent-attachment__image-frame .agent-attachment__preview { width: 100%; height: 100%; object-fit: contain; margin: 0; } .agent-attachment__error { display: block; width: 100%; color: #fda4af; font-size: 12px; margin-top: 5px; } diff --git a/web/src/components/agent/agent-conversation.tsx b/web/src/components/agent/agent-conversation.tsx index e1e09a1..988d46e 100644 --- a/web/src/components/agent/agent-conversation.tsx +++ b/web/src/components/agent/agent-conversation.tsx @@ -61,8 +61,8 @@ const AgentBubble = memo(function AgentBubble({ message, animate, onRetry, downl {!own ? : null}
- {own ?

{message.content}

: } - {agentAttachments(message.metadata).map(file => )} + {own ?

{message.content}

: } + {agentAttachments(message.metadata, message.role).map(file => )}
{own && failed && typeof info.error === "string" ?
{info.error}
: null}
diff --git a/web/src/components/markdown/markdown-content.tsx b/web/src/components/markdown/markdown-content.tsx index d28baf5..376af55 100644 --- a/web/src/components/markdown/markdown-content.tsx +++ b/web/src/components/markdown/markdown-content.tsx @@ -11,6 +11,8 @@ type Props = { mode?: MarkdownMode; resolveCitationIndex?: (href: string) => number | undefined; openLinksInNewTab?: boolean; + /** Agent images are delivered only through authenticated attachment IDs. */ + allowImages?: boolean; }; function replaceMarkdownLinksWithCitationTokens(markdown: string, resolveCitationIndex?: (href: string) => number | undefined) { @@ -35,24 +37,28 @@ function setNewTabLinkAttributes(currentNode: Element) { currentNode.setAttribute("rel", "noopener noreferrer"); } -function renderMarkdown(markdown: string, openLinksInNewTab: boolean) { +function renderMarkdown(markdown: string, openLinksInNewTab: boolean, allowImages: boolean) { const rawHtml = marked.parse(markdown, { gfm: true, breaks: true, renderer: markdownRenderer }) as string; - if (!openLinksInNewTab) return DOMPurify.sanitize(rawHtml, { ADD_ATTR: ["class", "target", "rel"] }); + const config = { ADD_ATTR: ["class", "target", "rel"], ...(!allowImages ? { + FORBID_TAGS: ["img", "picture", "source", "video", "audio", "svg", "math", "style"], + FORBID_ATTR: ["style", "srcset", "background", "poster"], + } : {}) }; + if (!openLinksInNewTab) return DOMPurify.sanitize(rawHtml, config); DOMPurify.addHook("afterSanitizeAttributes", setNewTabLinkAttributes); try { - return DOMPurify.sanitize(rawHtml, { ADD_ATTR: ["class", "target", "rel"] }); + return DOMPurify.sanitize(rawHtml, config); } finally { DOMPurify.removeHook("afterSanitizeAttributes", setNewTabLinkAttributes); } } -export function MarkdownContent({ markdown, className, mode = "default", resolveCitationIndex, openLinksInNewTab = false }: Props) { +export function MarkdownContent({ markdown, className, mode = "default", resolveCitationIndex, openLinksInNewTab = false, allowImages = true }: Props) { const html = useMemo(() => { const prepared = mode === "citationTokens" ? replaceMarkdownLinksWithCitationTokens(markdown, resolveCitationIndex) : markdown; - return renderMarkdown(prepared, openLinksInNewTab); - }, [markdown, mode, openLinksInNewTab, resolveCitationIndex]); + return renderMarkdown(prepared, openLinksInNewTab, allowImages); + }, [markdown, mode, openLinksInNewTab, resolveCitationIndex, allowImages]); return
; } diff --git a/web/src/lib/agent-attachments.ts b/web/src/lib/agent-attachments.ts index 681b507..8e2c87a 100644 --- a/web/src/lib/agent-attachments.ts +++ b/web/src/lib/agent-attachments.ts @@ -19,10 +19,23 @@ export function parseAgentAttachment(value: unknown): AgentAttachment { return { id: item.id, kind: item.kind, filename: item.filename, mimeType: item.mimeType, sizeBytes: item.sizeBytes, sha256: item.sha256, ...(item.workspacePath ? { workspacePath: item.workspacePath } : {}) }; } -export function agentAttachments(metadata: unknown): AgentAttachment[] { +export function agentAttachments(metadata: unknown, role = "user"): AgentAttachment[] { + if (role !== "user" && role !== "assistant") return []; const values = (metadata as { agentAttachments?: unknown } | null)?.agentAttachments; if (!Array.isArray(values) || values.length > 8) return []; - try { return values.map(parseAgentAttachment); } catch { return []; } + try { + const files = values.map(value => { + // Source paths are not part of the assistant output contract, even if an + // older/malformed server supplies one. Only the ID authorizes retrieval. + const file = parseAgentAttachment(role === "assistant" ? { ...value, workspacePath: undefined } : value); + if (role === "assistant" && (file.kind !== "image" || !["image/png", "image/jpeg", "image/gif", "image/webp"].includes(file.mimeType) + || file.sizeBytes > 6 * 1024 * 1024 || file.filename.length > 255 || /[\\/\u0000-\u001f\u007f]/.test(file.filename))) + throw new Error("Invalid assistant image."); + return file; + }); + if (new Set(files.map(file => file.id)).size !== files.length || files.reduce((total, file) => total + file.sizeBytes, 0) > 32 * 1024 * 1024) return []; + return files; + } catch { return []; } } export function attachmentCaption(content: string, files: AgentAttachment[]): string { return content.trim() ? content : `Attached: ${files.map(file => file.filename).join(", ")}.`; diff --git a/web/src/lib/agent-room-presentation.ts b/web/src/lib/agent-room-presentation.ts index a07feee..e9378e6 100644 --- a/web/src/lib/agent-room-presentation.ts +++ b/web/src/lib/agent-room-presentation.ts @@ -20,7 +20,7 @@ export type AgentRoomPresenterOptions = { typingDelayMs?: number; clock?: Scheduler; }; -const quietTools = new Set(['user_message', 'notify_parent', 'clock_wait', 'event_decision']); +const quietTools = new Set(['user_message', 'output_image', 'notify_parent', 'clock_wait', 'event_decision']); const data = (message: Message): Record => message.metadata && typeof message.metadata === 'object' && !Array.isArray(message.metadata) ? message.metadata as Record : {}; const entryId = (message: Message): bigint | undefined => { diff --git a/web/src/lib/agent-room.ts b/web/src/lib/agent-room.ts index 97c4b33..0f59f8f 100644 --- a/web/src/lib/agent-room.ts +++ b/web/src/lib/agent-room.ts @@ -18,7 +18,7 @@ export type AgentMessagesPage = { }; export type AgentDescriptor = { id: string; title: string; model: string; activity: AgentActivity; - websocket: string; protocol: string; capabilities?: { attachments?: AgentAttachmentCapability }; + websocket: string; protocol: string; capabilities?: { attachments?: AgentAttachmentCapability; assistantImages?: { version: 1 } }; }; export type AgentReadState = { cursor: string; updatedAt: string | null; sourceId?: string | null }; export type AgentRoomState = { @@ -175,6 +175,8 @@ export class AgentRoomClient { private readonly baseUrl: string; private readonly fetcher: typeof globalThis.fetch; private readonly token: () => string | null; + private activeDownloads = 0; + private readonly downloadWaiters = new Set<() => void>(); constructor(options: AgentRoomClientOptions = {}) { this.baseUrl = (options.baseUrl || "/api").replace(/\/+$/, ""); this.fetcher = (options.fetch || globalThis.fetch).bind(globalThis); @@ -223,12 +225,46 @@ export class AgentRoomClient { } async download(attachment: AgentAttachment, signal?: AbortSignal): Promise { const file = parseAgentAttachment(attachment), headers = new Headers(), token = this.token(); - if (token) headers.set("Authorization", `Bearer ${token}`); - const response = await this.fetcher(`${this.baseUrl}/v1/agent/attachments/${encodeURIComponent(file.id)}/content`, { headers, signal, cache: "no-store", redirect: "error" }); - if (!response.ok) { const error = await response.json().catch(() => null); throw new Error(error?.message || "Attachment download failed."); } - const blob = await response.blob(); - if (blob.size !== file.sizeBytes || blob.size > 8 * 1024 * 1024) throw new Error("Attachment download was incomplete."); - return blob; + signal?.throwIfAborted(); + while (this.activeDownloads >= 2) { + await new Promise((resolve, reject) => { + const cleanup = () => { this.downloadWaiters.delete(wake); signal?.removeEventListener("abort", abort); }; + const wake = () => { cleanup(); resolve(); }; + const abort = () => { cleanup(); reject(signal?.reason || new Error("Download canceled.")); }; + this.downloadWaiters.add(wake); signal?.addEventListener("abort", abort, { once: true }); + if (signal?.aborted) abort(); + }); + signal?.throwIfAborted(); + } + this.activeDownloads++; + try { + if (token !== this.token()) throw new Error("Authentication changed. Retry the download."); + if (token) headers.set("Authorization", `Bearer ${token}`); + const response = await this.fetcher(`${this.baseUrl}/v1/agent/attachments/${encodeURIComponent(file.id)}/content`, { headers, signal, cache: "no-store", redirect: "error" }); + if (!response.ok) { await response.body?.cancel(); throw new Error("Attachment download failed. Retry the download."); } + const reader = response.body?.getReader(); + if (!reader) throw new Error("Attachment download was incomplete."); + const bytes = new Uint8Array(file.sizeBytes); let size = 0; + try { + for (;;) { + const { done, value } = await reader.read(); if (done) break; + signal?.throwIfAborted(); + if (size + value.length > bytes.length) throw new Error("Attachment download exceeds its size limit."); + bytes.set(value, size); size += value.length; + } + } finally { await reader.cancel().catch(() => {}); reader.releaseLock(); } + if (size !== file.sizeBytes) throw new Error("Attachment download was incomplete."); + const sha256 = [...new Uint8Array(await crypto.subtle.digest("SHA-256", bytes))].map(n => n.toString(16).padStart(2, "0")).join(""); + if (sha256 !== file.sha256) throw new Error("Attachment checksum did not match."); + if (file.kind === "image" && response.headers.get("content-type")?.split(";")[0].trim() !== file.mimeType) + throw new Error("Attachment image type did not match."); + signal?.throwIfAborted(); + if (token !== this.token()) throw new Error("Authentication changed. Retry the download."); + return new Blob([bytes], { type: response.headers.get("content-type") || "application/octet-stream" }); + } finally { + this.activeDownloads--; + this.downloadWaiters.values().next().value?.(); + } } websocketUrl(pageUrl: string): string { const url = new URL(`${this.baseUrl}/v1/agent/ws`, pageUrl); diff --git a/web/tests/agent-attachments.test.mjs b/web/tests/agent-attachments.test.mjs index 89a3b44..bd8fe8f 100644 --- a/web/tests/agent-attachments.test.mjs +++ b/web/tests/agent-attachments.test.mjs @@ -5,11 +5,48 @@ import { createAgentUploadQueue, agentAttachments, parseAgentAttachment } from ' import { AgentRoomClient } from '../src/lib/agent-room.ts'; import { createAgentOptimisticOutbox } from '../src/lib/agent-optimistic-outbox.ts'; import { agentTranscriptMessages } from '../src/lib/agent-room-presentation.ts'; +import { createAgentRoomState, applyAgentEvent, mergeAgentMessages } from '../src/lib/agent-room.ts'; const file = { id:'att_12345678-1234-4234-8234-123456789abc',kind:'text',filename:'notes.txt',mimeType:'text/plain',sizeBytes:4,sha256:createHash('sha256').update('data').digest('hex'),workspacePath:'/data/workspace/uploads/att_12345678-1234-4234-8234-123456789abc/notes.txt' }; const cap = {version:1,maxCount:8,maxTotalBytes:32*1024*1024,maxImageBytes:6*1024*1024,maxFileBytes:8*1024*1024,imageMimeTypes:['image/png','image/jpeg','image/gif','image/webp']}; const tick=()=>new Promise(r=>setTimeout(r,0)); const waitFor=async check=>{const until=Date.now()+5000;while(!check()){assert.ok(Date.now(){ + const image={...file,kind:'image',mimeType:'image/png',filename:'image.png',url:'https://never-fetch.invalid/image'}; + const safe={...image};delete safe.workspacePath;delete safe.url; + const message={id:'outbound:42',role:'assistant',content:'',createdAt:new Date().toISOString(),metadata:{kind:'message',status:'complete',entryId:42,agentAttachments:[image]}}; + const page={type:'messages',conversationId:'1',messages:[message],cursor:'42',beforeCursor:'42',hasMore:false}; + const live=applyAgentEvent(createAgentRoomState(),page); + assert.deepEqual(agentAttachments(live.messages[0].metadata,'assistant'),[safe]); + const history=mergeAgentMessages(live.messages,[structuredClone(message)]); + assert.equal(history.length,1);assert.equal(history[0].id,'outbound:42'); + assert.deepEqual(agentAttachments(history[0].metadata,'assistant'),[safe]); + assert.equal(agentTranscriptMessages(history).length,1,'image-only assistant bubble remains visible'); + for(const invalid of [{...image,kind:'file'},{...image,mimeType:'image/svg+xml'},{...image,sizeBytes:6*1024*1024+1},{...image,filename:'/private/source.png'}]) + assert.deepEqual(agentAttachments({agentAttachments:[invalid]},'assistant'),[]); + assert.deepEqual(agentAttachments(message.metadata,'tool'),[]); + assert.deepEqual(agentAttachments({agentAttachments:[image,image]},'assistant'),[]); + const descriptor={capabilities:{assistantImages:{version:1}}}; + const queue=createAgentUploadQueue({capability:()=>descriptor.capabilities.attachments,changed:()=>{},upload:async()=>{throw new Error('Must not upload');}}); + assert.throws(()=>queue.add([new File(['data'],'notes.txt')]),/not available/);queue.dispose(); +}); + +test('downloads bound concurrent bytes, validate hashes/types, cancel queued work, and reject changed credentials',async()=>{ + const complete=[];let calls=0,token='one'; + const client=new AgentRoomClient({token:()=>token,fetch:async()=>{calls++;return new Promise(resolve=>complete.push(resolve));}}); + const a=client.download(file), b=client.download(file), cancel=new AbortController(); + const canceled=client.download(file,cancel.signal);const canceledAssertion=assert.rejects(canceled);cancel.abort(); + await canceledAssertion;assert.equal(calls,2); + complete.shift()(new Response('data'));await a; + const c=client.download(file);assert.equal(calls,3); + complete.shift()(new Response('evil'));await assert.rejects(b,/checksum/); + token='two';complete.shift()(new Response('data'));await assert.rejects(c,/Authentication changed/); + const huge=new AgentRoomClient({fetch:async()=>new Response('data-exceeds-receipt')}); + await assert.rejects(huge.download(file),/size limit/); + const wrongType=new AgentRoomClient({fetch:async()=>new Response('data',{headers:{'content-type':'text/html'}})}); + await assert.rejects(wrongType.download({...file,kind:'image',mimeType:'image/png'}),/type did not match/); +}); + test('composer gates unsupported servers, count/type/size and batches before upload; preserves retry ID and remove races',async()=>{ let capability, attempts=[], next=0, changed=0, finish; const queue=createAgentUploadQueue({capability:()=>capability,newId:()=>`upload-${++next}`,changed:()=>changed++,upload:async(_file,id,signal)=>{attempts.push({id,signal});if(attempts.length===1)throw new Error('Quota full');return new Promise(resolve=>finish=resolve);}});