JavaScript Media Source Table
| Piece | What it does | Field note |
|---|---|---|
new MediaSource() | The feed bag | attach via URL.createObjectURL, then sourceopen fires - all setup lives there; before it the object is closed and addSourceBuffer throws |
MediaSource.isTypeSupported(m) | The honest probe | full codec strings only (video/mp4; codecs=avc1.42E01E) - bare container names are invalid SourceBuffer types |
addSourceBuffer(codecs) | The track lane | one buffer per track, exact codec strings; mismatched audio/video codecs = the black-screen bug |
appendBuffer(data) | The feed cycle | async and serialized: one at a time, await updateend, guard sb.updating; init segment strictly first or the rest is refused |
buffered TimeRanges | The map | disjoint intervals of decodable data - query ranges vs currentTime, detect gaps, seek across them or refetch |
remove(start, end) | The pruning | drop old segments on live streams or the browser evicts for you under quota - silent start-trims kill DVR-style history |
endOfStream() | The finish line | signals no more data; skipping it leaves the element waiting forever on an edge that never comes |
fragmented media only | The hard rule | MSE takes fMP4/WebM clusters, not whole MP4 files - a valid file that refuses to play is a container problem, not a code bug |
Media Source Extensions (MSE) is the API that lets JavaScript feed a video element byte by byte: instead of pointing the player at a file URL, you create a MediaSource, attach it to the video via URL.createObjectURL, open one or more SourceBuffers for your audio and video tracks, and append encoded media segments as you have them. That one indirection is what makes adaptive bitrate streaming, live edges, ad insertion and pre-buffering possible in the browser - every major streaming player (hls.js, dash.js, Shaka) is an MSE client with a manifest parser on top.
Bottom line: the object model is small and strict. A MediaSource holds SourceBuffers - one per track, created with a codec string the browser must accept (sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs=vp9') - unsupported strings throw, so MediaSource.isTypeSupported() is your probe, the same honest pattern as MediaRecorder). Appends are async and serialized: write into buffer via appendBuffer(ArrayBuffer), wait for the updateend event, and never append to a buffer that is still updating (updating === true means queued work in flight). The buffer is a timeline of timeRanges (buffered.start(i)/end(i)), not a file - you query where you have data, append where you need it, and explicitly removeRange() old segments or the browser evicts for you under quota pressure.
The mental model that prevents every beginner bug: you are the demuxer feeder. MSE does not parse containers like MP4 wholesale - it needs fragmented media (fMP4, WebM clusters), where each append is a self-contained box carrying initialization metadata first (the init segment must be the first append or the buffer refuses the rest). Your job is fetch segments (range requests, HLS playlists, DASH manifests), order them, and keep the playhead ahead of the download edge - the browser handles decode, render, and the video element state machine.
How to use
- Wire the source: const ms = new MediaSource(); video.src = URL.createObjectURL(ms); ms.addEventListener('sourceopen', setup); - sourceopen is where all setup happens; before it fires the object is closed and addSourceBuffer throws.
- Probe and add buffers: if (MediaSource.isTypeSupported('video/mp4; codecs=avc1.42E01E, mp4a.40.2')) - create one SourceBuffer per track with the exact codec string; mismatched codecs between tracks is a black-screen bug with a console error nobody reads.
- Append in order: sb.addEventListener('updateend', appendNext); sb.appendBuffer(seg); - one append at a time per buffer, init segment strictly first, and guard with if (sb.updating) queue it; concurrent appends throw.
- Serve the playhead: check ms.sourceBuffers[i].buffered ranges against video.currentTime - fetch the segment covering (currentTime + lookahead); when the gap between buffered ranges swallows the playhead the video stalls, so detect gaps and seek across them.
- Prune and close: sb.remove(start, end) (await updateend) drops old segments on live streams; when done, ms.endOfStream() signals no more data and detaching means URL.revokeObjectURL - skipping endOfStream leaves the element waiting forever on an edge that never comes.
Frequently asked questions
Why does my video refuse to play when I append a valid MP4 file?
Because MSE does not accept whole files - it accepts fragmented media, and a regular MP4 is not fragmented. A standard MP4 puts its index (the moov atom) at the front or back of one contiguous file; MSE needs fMP4, where the index is spread through the stream as separate boxes so each append is independently meaningful. The failure looks like: init segment appends fine, media segments throw QuotaExceededError or silently never decode. Three practical exits. Use media already produced for streaming (HLS/DASH packagers like ffmpeg with -movflags +frag_keyframe+empty_moov emit fMP4). Use WebM, which is natively clustered and MSE-friendly (that is why every MSE tutorial uses Big Buck Bunny WebM). Or remux on the fly - muxer libraries (mux.js for TS-to-fMP4) exist precisely because the browser will not demux MPEG-TS for you. The codec string matters as much as the container: 'video/mp4' alone is not a valid SourceBuffer type; it needs the full codecs parameter, and isTypeSupported is the only honest check.
How do buffered ranges work, and why does my video stall mid-playback?
buffered is a TimeRanges object: a list of disjoint intervals where you have decodable data (buffered.start(i), buffered.end(i), buffered.length). Appends extend or create ranges; removes punch holes. Stalls come from four gap scenarios. Append gap: your segment fetch failed and the next append landed beyond the hole - the playhead walks into the gap and fires waiting; detect it (currentTime inside no range) and seek past or refetch. Edge chasing: you fetch too slowly and the playhead reaches the append frontier - widen your lookahead (3-4 segments typical) and prioritize the segment at the playhead over background quality switches. Eviction surprise: under quota pressure the browser trims the start of your buffer - a live stream that never calls remove() eventually appends into an error; prune explicitly (remove from 0 to currentTime - window). And codec-boundary artifacts: switching renditions mid-stream can create tiny unplayable slivers between ranges; players buffer across rendition switches at segment boundaries for exactly this reason. The diagnostic that answers everything: log buffered ranges every updateend, and the stall explains itself.
What are the quota and eviction rules for SourceBuffers?
MSE buffers live in a quota-managed tier, and the rules are engine-specific enough that you must design for eviction, not against it. The pressure signals: Chrome-family keeps roughly 100-300MB per origin for MSE (larger on desktop, and it shares the storage pool the Storage API describes); when you approach it, appends start failing with QuotaExceededError - which is a signal to prune, not to panic. The eviction behaviors: browsers may silently drop data from the start of your buffered range (oldest first - fine for live, fatal for a DVR UI that assumes history), and Safari historically evicted more aggressively than Chromium. The professional pattern: treat your own pruning as policy - remove(0, playhead - keepWindow) on every updateend for live streams, keep explicit windows for catch-up TV, and handle QuotaExceededError by removing the oldest quarter of the buffer and retrying the append. One more: appendBuffer data is copied out synchronously from the ArrayBuffer you pass - reusing or transferring that buffer before updateend corrupts nothing in the buffer but will corrupt whatever YOUR code expected to read back, so hand appends fresh copies or track ownership carefully.
When do I need MSE instead of a plain video tag - and when is it the wrong tool?
Plain src wins until you need control the element never exposes. The MSE triggers: adaptive bitrate (switch renditions on bandwidth - impossible with a static src), live streaming with a rolling window, pre-buffering specific segments (ad insertion, trick play), custom protocols your CDN speaks, or unified playback across DRM-free formats. If none of those apply, a plain video with preload and a CDN behind it is fewer moving parts and fewer failure modes. When MSE is the wrong tool even for streaming: DRM - Encrypted Media Extensions is a separate API that layers onto MSE (encrypted event, license negotiation), and managed content usually means EME plus a CDM, where hand-rolling MSE means hand-rolling license logic too. WebCodecs is the deeper wrong-tool boundary: it exposes decode frames without any video element at all - for analysis, transcoding pipelines, or rendering to canvas, skip MSE entirely. And the maturity note that saves projects: if you are building playback (not studying it), adopt hls.js or dash.js - they are MSE clients that have already absorbed the fragmented-container, gap-stall and eviction lessons above, and writing your own is a multi-quarter project disguised as a weekend.