buzzert/Sybil-2
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: extract text from fetched PDFs

Browse Source
This commit is contained in:
buzzert
2026-11-01 14:20:30 -08:00
parent a390c7287a
commit cb9d5a787b
13 changed files with 768 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/api/rest.md
+1 -1
View File
@@ -359,7 +359,7 @@ Behavior notes:
- Available Sybil-managed tool calls for `openai`, `anthropic`, `xai`, and `gemini`: `web_search` and `fetch_url`. When `CHAT_CODEX_TOOL_ENABLED=true`, `codex_exec` is also available. When `CHAT_SHELL_TOOL_ENABLED=true`, `shell_exec` is also available.
- `web_search` returns ranked results with per-result summaries/snippets. Its backend engine is selected by `CHAT_WEB_SEARCH_ENGINE`: `exa` (default), `brave` (requires `BRAVE_SEARCH_API_KEY`), or `searxng` (requires `SEARXNG_BASE_URL`; the instance must allow `format=json`).
- Brave searches are queued and evenly paced according to the shortest window in Brave's `X-RateLimit-Policy` response header. The backend also honors `X-RateLimit-Remaining`/`X-RateLimit-Reset` and retries `429` responses up to three times with reset-aware exponential backoff; quota resets beyond the bounded retry window fail immediately.
- `fetch_url` fetches a URL with browser-like navigation headers and returns plaintext page content (HTML converted to text server-side).
- `fetch_url` fetches an HTTP(S) URL with browser-like navigation headers and returns plaintext page content. HTML is converted to text. PDF files are detected by media type or PDF signature and have selectable text extracted with page markers (no OCR); at most 10 MiB and 30 pages are read per call. The optional `pages` array selects up to 30 page numbers for follow-up reads, and `maxCharacters` bounds PDF text collection; a brief truncation hint may follow the extracted text. PDF parsing runs in a worker terminated after 8 seconds. Responses identify page count, selected pages, truncation, encrypted PDFs, empty PDFs, and PDFs without extractable text; unavailable extraction returns `ok:false` with `pdfStatus`. Other URL responses also have a 10 MiB download limit.
- `codex_exec` delegates coding, shell, repository inspection, and other complex software tasks to a persistent remote Codex CLI workspace over SSH. The server runs `codex exec --dangerously-bypass-approvals-and-sandbox --skip-git-repo-check <non-interactive wrapped prompt>` on the configured devbox inside `CHAT_CODEX_REMOTE_WORKDIR`, with SSH stdin closed.
- `shell_exec` runs arbitrary non-interactive shell commands on the same configured devbox, starting in `CHAT_CODEX_REMOTE_WORKDIR`. It uses `bash -lc` when bash exists, otherwise `sh -lc`, closes SSH stdin, and does not run inside the Sybil server container.
- Devbox tool configuration: