carplay/AGENTS.md

AGENTS.md

This project is a web video player for clients that can decode audio and still images, but cannot use browser video decoding. Preserve that constraint when iterating: the browser should not use <video> playback for the source stream.

Product Shape

The UI intentionally has only two screens:

• URL entry screen with a stream URL input, a Play button, a Queue button, an env-gated Play Local button when LOCAL_VIDEOS is set, globally stored recently played URLs, and globally stored favorites.
• Fullscreen player screen with JPEG frames drawn to a canvas and native audio playback through an <audio> element.
• Playback controls are overlay controls toggled by tapping/clicking the frame area, similar to YouTube.
• Do not reintroduce debug panels, frame counters, settings forms, explanatory marketing copy, or visible ffmpeg details into the normal UI.

The backend stores recently played URLs and favorites globally, not per-browser. The default recent URL path is data/recent-urls.json, configurable with RECENT_URLS_PATH. The default favorites path is data/favorites.json, configurable with FAVORITES_PATH. Docker Compose persists both through the frame-stream-data volume.

Core Architecture

The app is plain Node/Express plus browser JavaScript:

• server/index.js: API, WebSocket, source proxy/relay, ffmpeg process lifecycle, recent URL and favorites persistence.
• public/index.html: frontend markup.
• public/app.js: URL submission, WebSocket frame receiving, audio element coordination, canvas drawing, overlay controls.
• public/styles.css: two-screen player UI.
• Dockerfile: production image with Node and ffmpeg.
• docker-compose-example.yml: operational example and default env knobs.

Main public endpoints:

• POST /api/session: validates the stream URL, stores recent URL, creates a short-lived playback session. Also accepts localPath for files selected from LOCAL_VIDEOS; local selections are not stored in recents.
• GET /api/local-videos: when LOCAL_VIDEOS is set, returns the recursive local file picker list.
• GET /api/recent-urls: returns global recent URL entries with url, redacted displayUrl, and lastPlayedAt.
• POST /api/recent-urls: validates a stream URL and stores it globally without creating a playback session.
• GET /api/favorites: returns global favorite entries with title and url.
• PUT /api/favorites: replaces the global favorites list. Each favorite has a user-provided title and stream url.
• GET /audio/:sessionId: serves MP3 audio to the browser audio element.
• WS /frames/:sessionId: sends timed JPEG frame packets to the browser.
• GET /api/health: exposes basic health and active playback connection mode.

Internal endpoint:

• /_source/:token: short-lived local proxy used by ffmpeg in split and single modes. This keeps original source URLs and query tokens out of ffmpeg process args and lets the server log upstream open/close behavior.

Browser Playback Model

Audio is the playback clock. The server sends JPEG frames over WebSocket. Each binary frame packet is:

• First 8 bytes: little-endian float64 timestamp in seconds.
• Remaining bytes: one complete JPEG image.

The frontend decodes JPEGs with browser image APIs, queues frames, and paints frames whose timestamps are due relative to audio.currentTime. This means the browser decodes only audio and still images, not video.

Why JPEG Frames

Keep JPEG unless there is a measured reason to change it.

• PNG is usually too large for 24 fps video.
• GIF has poor quality, weak timing control, and awkward streaming behavior.
• JPEG is browser-native, streamable frame-by-frame, much smaller than PNG, and simple to parse with SOI/EOI markers.

The server currently emits MJPEG through ffmpeg image2pipe. Frame parsing is done by scanning for JPEG SOI 0xff 0xd8 and EOI 0xff 0xd9.

Playback Modes

Playback mode is selected by PLAYBACK_CONNECTION_MODE. The code also accepts legacy aliases through PLAYBACK_MODE.

Use these modes deliberately:

• split: Smoothest mode. Starts separate ffmpeg workers and separate upstream source connections for audio and frames. Use for normal files and servers that allow multiple active connections.
• relay: IPTV-oriented mode. Opens one upstream HTTP connection from Node, then tees compressed input bytes into separate audio and frame ffmpeg workers through stdin. This preserves one source connection while isolating audio and frame processing.
• single: Fallback mode. Opens one upstream connection and one ffmpeg process with both MP3 and MJPEG outputs. This avoids multiple source connections but can stutter because audio and frame outputs are coupled inside one ffmpeg process.

The regression history matters:

• split was smooth, but some IPTV servers ended streams early when ffmpeg opened multiple connections/ranges.
• single fixed the one-active-connection issue, but introduced stutter because audio output backpressure and frame generation shared one ffmpeg process.
• relay was added to combine one upstream connection with separate ffmpeg workers.

Default code behavior is split when no mode is set. The Compose example uses relay because it is the mode to try for IPTV streams.

Finite-duration sessions are treated as recorded video and seekable once duration metadata is available. Metadata probing defaults off for unknown/live URLs in relay to preserve one upstream connection, but local files, YouTube and X/Twitter URLs without resolved duration metadata, recognized media file extensions, and Jellyfin download/stream URLs are probed automatically. Likely recorded sources retry failed probes up to three times in the background, keeping metadata pending until they finish. The timeout grows by a factor of three per attempt, capped at 30 seconds (default 4, 12, then 30 seconds), with 1- and 2-second retry delays. The frontend polls for the full pending period, including after stream reconnects.

When configured for relay, recognized recorded files use split immediately, before duration is known. HTTP inputs for these files allow automatic range seeking from startup, including when FFMPEG_INPUT_SEEKABLE=0 is configured for live streams. This avoids reading an entire MP4 through a pipe to find trailing metadata. Other sessions with known duration also use split; unknown/live streams stay in relay mode.

YouTube and X/Twitter sessions use the shared yt-dlp resolver and split, with separate resolved video and audio URLs when needed, subject to SINGLE_CONNECTION_HOSTS protections. Recognize x.com and twitter.com, including their www, m, and mobile variants. X posts can return playlists even with --no-playlist; use --playlist-items 1 and unwrap the first entry, while preserving explicit /video/N selections. X/Twitter defaults to direct HTTPS formats (preferring combined MP4s), since the source proxy does not rewrite relative HLS segment URLs. Direct MP4 formats may omit codec names: only the explicit value none means a stream is absent. The audio proxy must use the audio URL and its headers; frames and probes use the video URL and its headers. Enable yt-dlp's Node runtime explicitly with --js-runtimes node:<process.execPath>. Use yt-dlp's finite, non-live duration metadata directly to avoid an extra ffprobe connection.

IPTV subscriptions such as Oceanstream permit only one stream per customer. SINGLE_CONNECTION_HOSTS lists protected hostnames, including their subdomains; these sessions always use relay, never metadata probing or split, even for recorded-file URLs or when global probing/split is enabled. Seeking and recovery must not bypass this rule. Unknown live streams with probing disabled must not gain a metadata probe during recovery. The deployed list also includes the configured IPTV aliases line.oscreen.cc and yeahtv.live.

ffmpeg Pipelines

All ffmpeg command builders live near the bottom of server/index.js.

Common HTTP input args:

• -hide_banner
• -nostdin
• -loglevel ${FFMPEG_LOG_LEVEL}
• -nostats
• -seekable ${FFMPEG_INPUT_SEEKABLE}
• HTTP reconnect options when FFMPEG_HTTP_RECONNECT=1 and the input is HTTP(S)
• -re
• -i <inputUrl>

Pipe input args for relay intentionally skip -seekable. Some ffmpeg builds reject -seekable on pipe:0 with Option seekable not found.

Audio output:

• Maps 0:a:0?
• Disables video with -vn
• Converts to MP3 with libmp3lame
• Uses session.options.audioBitrate, default 160k
• Uses session.options.audioChannels, default 2
• Uses session.options.audioSampleRate, default 48000
• Outputs to pipe:1

Frame output:

• Maps 0:v:0
• Disables audio with -an
• Applies fps=<fps>,scale=w='min(<width>,iw)':h=-2:flags=bicubic:out_range=pc,format=yuvj420p,realtime
• Encodes mjpeg
• Uses -pix_fmt yuvj420p, -color_range pc, -q:v <quality>
• Outputs image2pipe to either pipe:1 or pipe:3

The explicit yuvj420p/full-range settings match the Docker image ffmpeg behavior. Older ffmpeg builds may still emit repeated swscaler warnings about deprecated pixel format. The logger suppresses only that known noisy warning so it cannot flood Docker logs and starve useful work.

Relay Mode Details

Relay mode is implemented by createRelayPlayback(session).

Important behavior:

• Waits until both audio HTTP response and frame WebSocket are attached.
• Starts two ffmpeg workers with pipe:0 input, one for audio and one for frames.
• Fetches the original session URL exactly once from Node, not through /_source/:token.
• Writes each upstream compressed chunk to both ffmpeg stdin streams.
• Uses bounded branch queues via createRelayInputBranch.
• Pauses upstream reading while any branch queue exceeds half of MAX_RELAY_BRANCH_QUEUE_BYTES.
• Stops playback if any branch queue exceeds MAX_RELAY_BRANCH_QUEUE_BYTES.
• Backpressure accounting must include both chunks queued in JavaScript and bytes already written to ffmpeg stdin while waiting for drain. Otherwise fast movie sources can outrun realtime ffmpeg consumption and grow Node heap until OOM.
• When waiting for relay capacity, wait only on branches that are actually over the pause threshold. Including already-ready branches in a Promise.race can create an immediate-resolution spin loop.

Relay mode works best for sequential stream containers such as MPEG-TS/IPTV. It may be less reliable for file formats that require seeking or late metadata, such as some MP4 files.

Cleanup Requirements

ffmpeg cleanup is important. Keep these invariants:

• If the audio client disconnects, stop the active playback for single and relay.
• If the frame WebSocket disconnects, stop the active playback for single and relay.
• In split, audio and frame workers are independent and each worker should stop when its own client side closes.
• Always release _source tokens when workers close.
• Always remove closed playbacks from the playbacks map.
• Use stopProcess(child): SIGTERM first, then SIGKILL after the timeout.
• Do not leave relay stdin streams open when stopping.

Useful local cleanup checks:

pgrep -af "ffmpeg.*pipe:0|ffmpeg.*_source"
pgrep -af "node server/index.js"

Both should be empty after smoke tests stop.

Logging

Operational logs are intended to be useful in Docker logs:

• ffmpeg process start/exit, PID, mode label, exit code, signal, duration.
• ffmpeg stderr lines except known swscaler pixel-format spam.
• source proxy connected/closed status, bytes, upstream end state.
• relay source connected/closed status, bytes, upstream end state.
• playback close summaries with frame counts, skipped frames, queue peaks.

Keep secrets redacted. redactSecrets currently redacts common query parameters such as api_key, apikey, access_token, token, and key.

Environment Knobs

Runtime:

• PORT: HTTP port, default 3000.
• FFMPEG_PATH: ffmpeg binary path, default ffmpeg.
• YT_DLP_PATH: yt-dlp binary path, default yt-dlp.
• YT_DLP_FORMAT: format selector override for YouTube and X/Twitter. YouTube defaults to bestvideo[protocol=https]+bestaudio[protocol=https]/bestvideo+bestaudio/best; X/Twitter defaults to best[protocol=https]/bestvideo[protocol=https]+bestaudio[protocol=https]. Separate audio/video inputs and combined sources are supported.
• YT_DLP_TIMEOUT_MS: yt-dlp resolution timeout, default 45000.
• FFMPEG_LOG_LEVEL: ffmpeg log level, default warning.
• FFMPEG_INPUT_SEEKABLE: HTTP input seekable option for unknown/live sources, default 0; recognized recorded sources use automatic range seeking.
• FFMPEG_HTTP_RECONNECT: enable ffmpeg HTTP reconnect options for HTTP inputs, default 1.
• FFMPEG_HTTP_RECONNECT_DELAY_MAX: max ffmpeg reconnect delay, default 2.
• FFMPEG_HTTP_RECONNECT_MAX_RETRIES: max ffmpeg reconnect retries, default 4, applied only when the installed ffmpeg supports reconnect_max_retries.
• FFMPEG_HTTP_RECONNECT_ON_HTTP_ERROR: HTTP status list for reconnect, default 5xx.
• METADATA_PROBE_ENABLED: probe session duration with ffprobe, default 1 except in relay, where default is 0.
• METADATA_PROBE_TIMEOUT_MS: initial ffprobe duration timeout, default 4000; recorded-source retry timeouts increase up to 30000.
• PLAYBACK_CONNECTION_MODE: split, relay, or single.
• SINGLE_CONNECTION_HOSTS: comma-separated subscription hosts that must use relay with no metadata probes or seeking; also matches their subdomains.
• RECENT_URLS_PATH: recent URL JSON path.
• RECENT_URL_LIMIT: recent URL count, default 12.
• FAVORITES_PATH: favorites JSON path.
• FAVORITES_LIMIT: favorites count, default 50.
• LOCAL_VIDEOS: optional local video directory. When set, the UI shows Play Local and lists regular files under this directory recursively.
• DEFAULT_FPS: default frame rate, fallback 24, clamped 1..30.
• DEFAULT_FRAME_WIDTH: default maximum frame width, fallback 960, clamped 160..1920.
• JPEG_QUALITY: default JPEG quality, fallback 7, clamped 2..18; lower is better for ffmpeg -q:v.
• DEFAULT_AUDIO_BITRATE: default MP3 audio bitrate, fallback 160k.
• DEFAULT_AUDIO_CHANNELS: default MP3 audio channels, fallback 2, clamped 1..2.
• DEFAULT_AUDIO_SAMPLE_RATE: default MP3 audio sample rate, fallback 48000, clamped 22050..48000.
• MAX_WS_BUFFER_BYTES: server-side WebSocket JPEG frame backlog cap, default 2097152.
• MAX_AUDIO_QUEUE_BYTES: single-mode audio output queue cap, default 4194304.
• MAX_RELAY_BRANCH_QUEUE_BYTES: relay per-branch compressed-input queue cap, default 8388608.

Session playback options are accepted by POST /api/session even though the UI hides them:

• fps: default 24, clamped 1..30.
• width: default 960, clamped 160..1920.
• quality: defaults to JPEG_QUALITY, clamped 2..18; lower is better for ffmpeg -q:v.
• audioBitrate: default 160k, accepts two or three digits followed by k.
• audioChannels: default 2, clamped 1..2.
• audioSampleRate: default 48000, clamped 22050..48000.

Docker Notes

The Docker image installs ffmpeg and yt-dlp and runs as non-root node. yt-dlp is installed from the upstream master branch when the image is built.

Hardware acceleration is not required. Device passthrough may help only if server CPU decode is saturated. It does not fix audio/frame coupling issues; relay was built for that.

Compose includes commented examples for:

• VAAPI passthrough through /dev/dri.
• NVIDIA passthrough with gpus: all.

Verification Commands

Basic validation:

node --check server/index.js
npm test
docker compose -f docker-compose-example.yml config
docker build -t frame-stream-player .

MPEG-TS local stream generation for playback smoke tests:

ffmpeg -y -hide_banner -loglevel error \
  -f lavfi -i testsrc2=size=320x180:rate=24 \
  -f lavfi -i sine=frequency=440:sample_rate=48000 \
  -t 8 \
  -c:v mpeg2video -pix_fmt yuv420p -b:v 900k \
  -c:a mp2 -b:a 128k \
  -f mpegts public/_relay-smoke.ts

Start a mode locally:

PORT=3014 RECENT_URLS_PATH=/tmp/carplay-relay-recent.json \
  FAVORITES_PATH=/tmp/carplay-relay-favorites.json \
  FFMPEG_LOG_LEVEL=warning FFMPEG_INPUT_SEEKABLE=0 \
  PLAYBACK_CONNECTION_MODE=relay npm start

After smoke testing, remove generated assets:

rm -f public/_relay-smoke.ts /tmp/carplay-*-recent.json /tmp/carplay-*-favorites.json

Security Notes

The server fetches arbitrary user-provided HTTP(S) URLs. Do not expose this app publicly without authentication and URL allowlisting or SSRF protections.

Do not log raw source URLs in normal operational logs. Use redaction for query-string secrets and prefer short internal _source URLs in ffmpeg args.

Change Guidance

Before changing the pipeline, decide which bottleneck you are addressing:

• Browser image decode or network bandwidth: lower width, lower fps, or increase JPEG quality number.
• Server CPU decode: consider ffmpeg tuning or hardware acceleration.
• Upstream server rejects multiple connections: use relay.
• Audio/frame stutter in one-connection mode: avoid single, use relay.
• Docker log floods: suppress only known noisy lines, not all stderr.

Avoid large frontend feature additions unless requested. The product goal is a minimal URL screen and a fullscreen player.