carplay/AGENTS.md

# AGENTS.md

This project is a web video player for clients that can decode audio and still images, but cannot use browser video decoding. Preserve that constraint when iterating: the browser should not use `<video>` playback for the source stream.

## Product Shape

The UI intentionally has only two screens:

- URL entry screen with a stream URL input, a `Play` button, a `Queue` button, an env-gated `Play Local` button when `LOCAL_VIDEOS` is set, globally stored recently played URLs, and globally stored favorites.
- Fullscreen player screen with JPEG frames drawn to a canvas and native audio playback through an `<audio>` element.
- Playback controls are overlay controls toggled by tapping/clicking the frame area, similar to YouTube.
- Do not reintroduce debug panels, frame counters, settings forms, explanatory marketing copy, or visible ffmpeg details into the normal UI.

The backend stores recently played URLs and favorites globally, not per-browser. The default recent URL path is `data/recent-urls.json`, configurable with `RECENT_URLS_PATH`. The default favorites path is `data/favorites.json`, configurable with `FAVORITES_PATH`. Docker Compose persists both through the `frame-stream-data` volume.

## Core Architecture

The app is plain Node/Express plus browser JavaScript:

- `server/index.js`: API, WebSocket, source proxy/relay, ffmpeg process lifecycle, recent URL and favorites persistence.
- `public/index.html`: frontend markup.
- `public/app.js`: URL submission, WebSocket frame receiving, audio element coordination, canvas drawing, overlay controls.
- `public/styles.css`: two-screen player UI.
- `Dockerfile`: production image with Node and ffmpeg.
- `docker-compose-example.yml`: operational example and default env knobs.

Main public endpoints:

- `POST /api/session`: validates the stream URL, stores recent URL, creates a short-lived playback session. Also accepts `localPath` for files selected from `LOCAL_VIDEOS`; local selections are not stored in recents.
- `GET /api/local-videos`: when `LOCAL_VIDEOS` is set, returns the recursive local file picker list.
- `GET /api/recent-urls`: returns global recent URL entries with `url`, redacted `displayUrl`, and `lastPlayedAt`.
- `POST /api/recent-urls`: validates a stream URL and stores it globally without creating a playback session.
- `GET /api/favorites`: returns global favorite entries with `title` and `url`.
- `PUT /api/favorites`: replaces the global favorites list. Each favorite has a user-provided `title` and stream `url`.
- `GET /audio/:sessionId`: serves MP3 audio to the browser audio element.
- `WS /frames/:sessionId`: sends timed JPEG frame packets to the browser.
- `GET /api/health`: exposes basic health and active playback connection mode.

Internal endpoint:

- `/_source/:token`: short-lived local proxy used by ffmpeg in split and single modes. This keeps original source URLs and query tokens out of ffmpeg process args and lets the server log upstream open/close behavior.

## Browser Playback Model

Audio is the playback clock. The server sends JPEG frames over WebSocket. Each binary frame packet is:

- First 8 bytes: little-endian float64 timestamp in seconds.
- Remaining bytes: one complete JPEG image.

The frontend decodes JPEGs with browser image APIs, queues frames, and paints frames whose timestamps are due relative to `audio.currentTime`. This means the browser decodes only audio and still images, not video.

## Why JPEG Frames

Keep JPEG unless there is a measured reason to change it.

- PNG is usually too large for 24 fps video.
- GIF has poor quality, weak timing control, and awkward streaming behavior.
- JPEG is browser-native, streamable frame-by-frame, much smaller than PNG, and simple to parse with SOI/EOI markers.

The server currently emits MJPEG through ffmpeg `image2pipe`. Frame parsing is done by scanning for JPEG SOI `0xff 0xd8` and EOI `0xff 0xd9`.

## Playback Modes

Playback mode is selected by `PLAYBACK_CONNECTION_MODE`. The code also accepts legacy aliases through `PLAYBACK_MODE`.

Use these modes deliberately:

- `split`: Smoothest mode. Starts separate ffmpeg workers and separate upstream source connections for audio and frames. Use for normal files and servers that allow multiple active connections.
- `relay`: IPTV-oriented mode. Opens one upstream HTTP connection from Node, then tees compressed input bytes into separate audio and frame ffmpeg workers through stdin. This preserves one source connection while isolating audio and frame processing.
- `single`: Fallback mode. Opens one upstream connection and one ffmpeg process with both MP3 and MJPEG outputs. This avoids multiple source connections but can stutter because audio and frame outputs are coupled inside one ffmpeg process.

The regression history matters:

- `split` was smooth, but some IPTV servers ended streams early when ffmpeg opened multiple connections/ranges.
- `single` fixed the one-active-connection issue, but introduced stutter because audio output backpressure and frame generation shared one ffmpeg process.
- `relay` was added to combine one upstream connection with separate ffmpeg workers.

Default code behavior is `split` when no mode is set. The Compose example uses `relay` because it is the mode to try for IPTV streams.

Finite-duration sessions are treated as recorded video and seekable once duration metadata is available. Metadata probing defaults off for unknown/live URLs in `relay` to preserve one upstream connection, but local files, YouTube and X/Twitter URLs without resolved duration metadata, recognized media file extensions, and Jellyfin download/stream URLs are probed automatically. Likely recorded sources retry failed probes up to three times in the background, keeping metadata pending until they finish. The timeout grows by a factor of three per attempt, capped at 30 seconds (default 4, 12, then 30 seconds), with 1- and 2-second retry delays. The frontend polls for the full pending period, including after stream reconnects.

When configured for `relay`, recognized recorded files use `split` immediately, before duration is known. HTTP inputs for these files allow automatic range seeking from startup, including when `FFMPEG_INPUT_SEEKABLE=0` is configured for live streams. This avoids reading an entire MP4 through a pipe to find trailing metadata. Other sessions with known duration also use `split`; unknown/live streams stay in relay mode.

YouTube and X/Twitter sessions use the shared yt-dlp resolver and `split`, with separate resolved video and audio URLs when needed, subject to `SINGLE_CONNECTION_HOSTS` protections. Recognize `x.com` and `twitter.com`, including their `www`, `m`, and `mobile` variants. X posts can return playlists even with `--no-playlist`; use `--playlist-items 1` and unwrap the first entry, while preserving explicit `/video/N` selections. X/Twitter defaults to direct HTTPS formats (preferring combined MP4s), since the source proxy does not rewrite relative HLS segment URLs. Direct MP4 formats may omit codec names: only the explicit value `none` means a stream is absent. The audio proxy must use the audio URL and its headers; frames and probes use the video URL and its headers. Enable yt-dlp's Node runtime explicitly with `--js-runtimes node:<process.execPath>`. Use yt-dlp's finite, non-live duration metadata directly to avoid an extra ffprobe connection.

IPTV subscriptions such as Oceanstream permit only one stream per customer. `SINGLE_CONNECTION_HOSTS` lists protected hostnames, including their subdomains; these sessions always use `relay`, never metadata probing or `split`, even for recorded-file URLs or when global probing/split is enabled. Seeking and recovery must not bypass this rule. Unknown live streams with probing disabled must not gain a metadata probe during recovery. The deployed list also includes the configured IPTV aliases `line.oscreen.cc` and `yeahtv.live`.

## ffmpeg Pipelines

All ffmpeg command builders live near the bottom of `server/index.js`.

Common HTTP input args:

- `-hide_banner`
- `-nostdin`
- `-loglevel ${FFMPEG_LOG_LEVEL}`
- `-nostats`
- `-seekable ${FFMPEG_INPUT_SEEKABLE}`
- HTTP reconnect options when `FFMPEG_HTTP_RECONNECT=1` and the input is HTTP(S)
- `-re`
- `-i <inputUrl>`

Pipe input args for relay intentionally skip `-seekable`. Some ffmpeg builds reject `-seekable` on `pipe:0` with `Option seekable not found`.

Audio output:

- Maps `0:a:0?`
- Disables video with `-vn`
- Converts to MP3 with `libmp3lame`
- Uses `session.options.audioBitrate`, default `160k`
- Uses `session.options.audioChannels`, default `2`
- Uses `session.options.audioSampleRate`, default `48000`
- Outputs to `pipe:1`

Frame output:

- Maps `0:v:0`
- Disables audio with `-an`
- Applies `fps=<fps>,scale=w='min(<width>,iw)':h=-2:flags=bicubic:out_range=pc,format=yuvj420p,realtime`
- Encodes `mjpeg`
- Uses `-pix_fmt yuvj420p`, `-color_range pc`, `-q:v <quality>`
- Outputs `image2pipe` to either `pipe:1` or `pipe:3`

The explicit `yuvj420p`/full-range settings match the Docker image ffmpeg behavior. Older ffmpeg builds may still emit repeated swscaler warnings about deprecated pixel format. The logger suppresses only that known noisy warning so it cannot flood Docker logs and starve useful work.

## Relay Mode Details

Relay mode is implemented by `createRelayPlayback(session)`.

Important behavior:

- Waits until both audio HTTP response and frame WebSocket are attached.
- Starts two ffmpeg workers with `pipe:0` input, one for audio and one for frames.
- Fetches the original session URL exactly once from Node, not through `/_source/:token`.
- Writes each upstream compressed chunk to both ffmpeg stdin streams.
- Uses bounded branch queues via `createRelayInputBranch`.
- Pauses upstream reading while any branch queue exceeds half of `MAX_RELAY_BRANCH_QUEUE_BYTES`.
- Stops playback if any branch queue exceeds `MAX_RELAY_BRANCH_QUEUE_BYTES`.
- Backpressure accounting must include both chunks queued in JavaScript and bytes already written to ffmpeg stdin while waiting for `drain`. Otherwise fast movie sources can outrun realtime ffmpeg consumption and grow Node heap until OOM.
- When waiting for relay capacity, wait only on branches that are actually over the pause threshold. Including already-ready branches in a `Promise.race` can create an immediate-resolution spin loop.

Relay mode works best for sequential stream containers such as MPEG-TS/IPTV. It may be less reliable for file formats that require seeking or late metadata, such as some MP4 files.

## Cleanup Requirements

ffmpeg cleanup is important. Keep these invariants:

- If the audio client disconnects, stop the active playback for `single` and `relay`.
- If the frame WebSocket disconnects, stop the active playback for `single` and `relay`.
- In `split`, audio and frame workers are independent and each worker should stop when its own client side closes.
- Always release `_source` tokens when workers close.
- Always remove closed playbacks from the `playbacks` map.
- Use `stopProcess(child)`: SIGTERM first, then SIGKILL after the timeout.
- Do not leave relay stdin streams open when stopping.

Useful local cleanup checks:

```sh
pgrep -af "ffmpeg.*pipe:0|ffmpeg.*_source"
pgrep -af "node server/index.js"
```

Both should be empty after smoke tests stop.

## Logging

Operational logs are intended to be useful in Docker logs:

- ffmpeg process start/exit, PID, mode label, exit code, signal, duration.
- ffmpeg stderr lines except known swscaler pixel-format spam.
- source proxy connected/closed status, bytes, upstream end state.
- relay source connected/closed status, bytes, upstream end state.
- playback close summaries with frame counts, skipped frames, queue peaks.

Keep secrets redacted. `redactSecrets` currently redacts common query parameters such as `api_key`, `apikey`, `access_token`, `token`, and `key`.

## Environment Knobs

Runtime:

- `PORT`: HTTP port, default `3000`.
- `FFMPEG_PATH`: ffmpeg binary path, default `ffmpeg`.
- `YT_DLP_PATH`: yt-dlp binary path, default `yt-dlp`.
- `YT_DLP_FORMAT`: format selector override for YouTube and X/Twitter. YouTube defaults to `bestvideo[protocol=https]+bestaudio[protocol=https]/bestvideo+bestaudio/best`; X/Twitter defaults to `best[protocol=https]/bestvideo[protocol=https]+bestaudio[protocol=https]`. Separate audio/video inputs and combined sources are supported.
- `YT_DLP_TIMEOUT_MS`: yt-dlp resolution timeout, default `45000`.
- `FFMPEG_LOG_LEVEL`: ffmpeg log level, default `warning`.
- `FFMPEG_INPUT_SEEKABLE`: HTTP input seekable option for unknown/live sources, default `0`; recognized recorded sources use automatic range seeking.
- `FFMPEG_HTTP_RECONNECT`: enable ffmpeg HTTP reconnect options for HTTP inputs, default `1`.
- `FFMPEG_HTTP_RECONNECT_DELAY_MAX`: max ffmpeg reconnect delay, default `2`.
- `FFMPEG_HTTP_RECONNECT_MAX_RETRIES`: max ffmpeg reconnect retries, default `4`, applied only when the installed ffmpeg supports `reconnect_max_retries`.
- `FFMPEG_HTTP_RECONNECT_ON_HTTP_ERROR`: HTTP status list for reconnect, default `5xx`.
- `METADATA_PROBE_ENABLED`: probe session duration with ffprobe, default `1` except in `relay`, where default is `0`.
- `METADATA_PROBE_TIMEOUT_MS`: initial ffprobe duration timeout, default `4000`; recorded-source retry timeouts increase up to `30000`.
- `PLAYBACK_CONNECTION_MODE`: `split`, `relay`, or `single`.
- `SINGLE_CONNECTION_HOSTS`: comma-separated subscription hosts that must use relay with no metadata probes or seeking; also matches their subdomains.
- `RECENT_URLS_PATH`: recent URL JSON path.
- `RECENT_URL_LIMIT`: recent URL count, default `12`.
- `FAVORITES_PATH`: favorites JSON path.
- `FAVORITES_LIMIT`: favorites count, default `50`.
- `LOCAL_VIDEOS`: optional local video directory. When set, the UI shows `Play Local` and lists regular files under this directory recursively.
- `DEFAULT_FPS`: default frame rate, fallback `24`, clamped `1..30`.
- `DEFAULT_FRAME_WIDTH`: default maximum frame width, fallback `960`, clamped `160..1920`.
- `JPEG_QUALITY`: default JPEG quality, fallback `7`, clamped `2..18`; lower is better for ffmpeg `-q:v`.
- `DEFAULT_AUDIO_BITRATE`: default MP3 audio bitrate, fallback `160k`.
- `DEFAULT_AUDIO_CHANNELS`: default MP3 audio channels, fallback `2`, clamped `1..2`.
- `DEFAULT_AUDIO_SAMPLE_RATE`: default MP3 audio sample rate, fallback `48000`, clamped `22050..48000`.
- `MAX_WS_BUFFER_BYTES`: server-side WebSocket JPEG frame backlog cap, default `2097152`.
- `MAX_AUDIO_QUEUE_BYTES`: single-mode audio output queue cap, default `4194304`.
- `MAX_RELAY_BRANCH_QUEUE_BYTES`: relay per-branch compressed-input queue cap, default `8388608`.

Session playback options are accepted by `POST /api/session` even though the UI hides them:

- `fps`: default `24`, clamped `1..30`.
- `width`: default `960`, clamped `160..1920`.
- `quality`: defaults to `JPEG_QUALITY`, clamped `2..18`; lower is better for ffmpeg `-q:v`.
- `audioBitrate`: default `160k`, accepts two or three digits followed by `k`.
- `audioChannels`: default `2`, clamped `1..2`.
- `audioSampleRate`: default `48000`, clamped `22050..48000`.

## Docker Notes

The Docker image installs ffmpeg and yt-dlp and runs as non-root `node`. yt-dlp is installed from the upstream master branch when the image is built.

Hardware acceleration is not required. Device passthrough may help only if server CPU decode is saturated. It does not fix audio/frame coupling issues; `relay` was built for that.

Compose includes commented examples for:

- VAAPI passthrough through `/dev/dri`.
- NVIDIA passthrough with `gpus: all`.

## Verification Commands

Basic validation:

```sh
node --check server/index.js
npm test
docker compose -f docker-compose-example.yml config
docker build -t frame-stream-player .
```

MPEG-TS local stream generation for playback smoke tests:

```sh
ffmpeg -y -hide_banner -loglevel error \
  -f lavfi -i testsrc2=size=320x180:rate=24 \
  -f lavfi -i sine=frequency=440:sample_rate=48000 \
  -t 8 \
  -c:v mpeg2video -pix_fmt yuv420p -b:v 900k \
  -c:a mp2 -b:a 128k \
  -f mpegts public/_relay-smoke.ts
```

Start a mode locally:

```sh
PORT=3014 RECENT_URLS_PATH=/tmp/carplay-relay-recent.json \
  FAVORITES_PATH=/tmp/carplay-relay-favorites.json \
  FFMPEG_LOG_LEVEL=warning FFMPEG_INPUT_SEEKABLE=0 \
  PLAYBACK_CONNECTION_MODE=relay npm start
```

After smoke testing, remove generated assets:

```sh
rm -f public/_relay-smoke.ts /tmp/carplay-*-recent.json /tmp/carplay-*-favorites.json
```

## Security Notes

The server fetches arbitrary user-provided HTTP(S) URLs. Do not expose this app publicly without authentication and URL allowlisting or SSRF protections.

Do not log raw source URLs in normal operational logs. Use redaction for query-string secrets and prefer short internal `_source` URLs in ffmpeg args.

## Change Guidance

Before changing the pipeline, decide which bottleneck you are addressing:

- Browser image decode or network bandwidth: lower `width`, lower `fps`, or increase JPEG `quality` number.
- Server CPU decode: consider ffmpeg tuning or hardware acceleration.
- Upstream server rejects multiple connections: use `relay`.
- Audio/frame stutter in one-connection mode: avoid `single`, use `relay`.
- Docker log floods: suppress only known noisy lines, not all stderr.

Avoid large frontend feature additions unless requested. The product goal is a minimal URL screen and a fullscreen player.