Skip to main content
Version: 1.0.0

WebCodecs Decoder

Decode and render H.264, H.265 and AV1 streams in Web browsers using WebCodecs API, the new Web standard for hardware-accelerated video decoding.

It's fast, uses less hardware resources, and supports more profiles and levels. However, it's only available in recent versions of Chrome and Safari.

npm install @yume-chan/scrcpy-decoder-webcodecs

Feature detection​

To check if the WebCodecs API is available, you can check if VideoDecoder is defined:

const isSupported = globalThis.VideoDecoder !== undefined;

VideoDecoder.isConfigSupported is a static method that checks if a given codec configuration is supported.

It accepts a codec parameter string, for example "hev1.1.60.L153.B0.0.0.0.0.0" for H.265 and "av01.0.05M.08" for AV1.

const result = await VideoDecoder.isConfigSupported({
  codec: "hev1.1.60.L153.B0.0.0.0.0.0",
});
const isHevcSupported = result.supported === true;

You can decide which video codec to use based on the results.

Renderer​

WebCodecs API decodes video frames into VideoFrame objects. There are multiple methods to render those VideoFrame objects onto the page. This package provides three renderers:

info

These renderers are not tied to our WebCodecsVideoDecoder, they can also be used separately to render any VideoFrame objects from WebCodecs API.

  • InsertableStreamWebCodecsDecoderRenderer: Renders to a <video> element using Insertable Streams API. See quirks below.
  • WebGLWebCodecsDecoderRenderer: Renders to a <canvas> or OffscreenCanvas using WebGL. It only works with hardware accelerated WebGL, because without hardware acceleration, the performance is even worse than the bitmap renderer below.
  • BitmapWebCodecsDecoderRenderer: Renders to a <canvas> or OffscreenCanvas using bitmap renderer.
info

VideoFrames can also be rendered using 2D canvas. However, because it's slower than bitmap renderer, and bitmap renderer is already available on all devices, we didn't think it's necessary to implement it.

Quirks of Insertable Stream renderer​

The Insertable Streams renderer should be considered as experimental, because there are several issues around it:

Performance​

The Insertable Streams API is specifically designed to render video frames from WebCodecs API, but in reality it's only easier to integrate, not faster. So it doesn't have the performance advantage over other renderers.

Compatibility​

Its specification has two versions: the old MediaStreamTrackGenerator API, and the new VideoTrackGenerator. Only Chrome implemented the old API. The new API was added in mid 2023, but until end of 2024, nobody (including Chrome, who authored the specification), has implemented the new API (Chrome issue, Firefox issue).

As a result, we implemented the Insertable Stream renderer using the old MediaStreamTrackGenerator API. We will monitor the situation and update the renderer if necessary.

Lifecycle​

Because it renders to a <video> element, if the video element is removed from the DOM tree (e.g. to move it into another element, or another page), it will be automatically paused. You need to call renderer.element.play() to resume playback after adding it back to the DOM tree.

It sets the autoplay attribute on the <video> element, so it will start playing automatically for the first time.

Create a renderer​

Generally, the performance ranking is InsertableStream ≈ WebGL >> Bitmap. However, because Insertable Stream renderer and WebGL renderer are not available on all devices, we recommend the following method to choose the best renderer on all devices:

InsertableStreamWebCodecsDecoderRenderer and WebGLWebCodecsDecoderRenderer both have an isSupported static property, to check whether they are supported by the current browser and hardware:

import type { WebCodecsVideoDecoderRenderer } from "@yume-chan/scrcpy-decoder-webcodecs";
import {
  InsertableStreamWebCodecsDecoderRenderer,
  WebGLWebCodecsDecoderRenderer,
  BitmapWebCodecsDecoderRenderer,
} from "@yume-chan/scrcpy-decoder-webcodecs";

function createWebCodecsRenderer(): {
  renderer: WebCodecsVideoDecoderRenderer;
  element: HTMLVideoElement | HTMLCanvasElement;
} {
  if (InsertableStreamWebCodecsDecoderRenderer.isSupported) {
    const renderer = new InsertableStreamWebCodecsDecoderRenderer();
    return { renderer, element: renderer.element };
  }

  if (WebGLWebCodecsDecoderRenderer.isSupported) {
    const renderer = new WebGLWebCodecsDecoderRenderer();
    return { renderer, element: renderer.canvas as HTMLCanvasElement };
  }

  const renderer = new BitmapWebCodecsDecoderRenderer();
  return { renderer, element: renderer.canvas as HTMLCanvasElement };
}

When the constructors are called without arguments, they will create a rendering target automatically (<video> element for InsertableStreamWebCodecsDecoderRenderer, <canvas> for WebGLWebCodecsDecoderRenderer and BitmapWebCodecsDecoderRenderer). You will need to insert the created element into the page to display the video:

const { renderer, element } = createWebCodecsRenderer();
document.body.appendChild(element);

Or, all renderers accept existing rendering targets:

new InsertableStreamWebCodecsDecoderRenderer(videoElement);
new WebGLWebCodecsDecoderRenderer(canvasElementOrOffscreenCanvas);
new BitmapWebCodecsDecoderRenderer(canvasElementOrOffscreenCanvas);

Create a decoder​

import type { ScrcpyMediaStreamPacket } from "@yume-chan/scrcpy";
import { ScrcpyVideoCodecId } from "@yume-chan/scrcpy";
import type {
  ScrcpyVideoDecoder,
  ScrcpyVideoDecoderCapability,
} from "@yume-chan/scrcpy-decoder-tinyh264";
import { WritableStream } from "@yume-chan/stream-extra";
import type { WebCodecsVideoDecoderRenderer } from "./render/index.js";

export declare class WebCodecsVideoDecoder implements ScrcpyVideoDecoder {
  static get isSupported(): boolean;
  static readonly capabilities: Record<string, ScrcpyVideoDecoderCapability>;

  get codec(): ScrcpyVideoCodecId;
  get writable(): WritableStream<ScrcpyMediaStreamPacket>;
  get renderer(): WebCodecsVideoDecoderRenderer;

  get framesRendered(): number;
  get framesSkipped(): number;

  get sizeChanged(): import("@yume-chan/event").AddEventListener<
    {
      width: number;
      height: number;
    },
    unknown
  >;

  /**
   * Create a new WebCodecs video decoder.
   */
  constructor(options: WebCodecsVideoDecoder.Options);

  snapshot(): Promise<Blob | undefined>;
  dispose(): void;
}

export declare namespace WebCodecsVideoDecoder {
  interface Options {
    /**
     * The video codec to decode
     */
    codec: ScrcpyVideoCodecId;

    renderer: WebCodecsVideoDecoderRenderer;
  }
}

The constructor requires two options:

  • codec: the video codec to be decoded. It can be retrieved from the video stream metadata, or hard-coded if you only use a specific video codec.
  • renderer: a renderer created in the previous section.

Similar to the TinyH264 decoder, after creating a decoder instance, you need to pipe the video stream into the writable stream:

import type { ScrcpyMediaStreamPacket } from "@yume-chan/scrcpy";
import { WebCodecsVideoDecoder } from "@yume-chan/scrcpy-decoder-webcodecs";

declare const videoPacketStream: ReadableStream<ScrcpyMediaStreamPacket>;

const decoder = new WebCodecsVideoDecoder({
  codec: videoMetadata.codec,
  renderer: renderer,
});

void videoPacketStream.pipeTo(decoder.writable).catch((e) => {
  console.error(e);
});

Handle size changes​

When the device orientation changes, Scrcpy server will recreate a new video encoder with the new size. The decoder will parse the new video configuration, and update the canvas size automatically.

However, the video size is also useful for other purposes, like injecting touch events. The sizeChanged event will be emitted when the video size changes:

decoder.sizeChanged(({ width, height }) => {
  console.log(width, height);
});

Use in Web Worker​

Although WebCodecs API already runs in its own thread, and the renderers are very fast, you might still want to run them in a dedicated Web Worker, so other works on the main thread won't affect the performance.

Only WebGLWebCodecsDecoderRenderer and BitmapWebCodecsDecoderRenderer are supported in Web Worker. There are two ways to render the frames:

  • When their constructors are called without arguments, they will create an OffscreenCanvas object. See Synchronous display of frames produced by an OffscreenCanvas on MDN for how to use it.
  • Call transferControlToOffscreen() on a <canvas> element, postMessage the OffscreenCanvas object to the worker, and use it to create the renderer.
<!-- index.html -->
<canvas id="canvas"></canvas>
// index.js
import type { ScrcpyMediaStreamPacket } from "@yume-chan/scrcpy";
import { ScrcpyVideoCodecId } from "@yume-chan/scrcpy";

declare const videoPacketStream: ReadableStream<ScrcpyMediaStreamPacket>;

const canvas = document.getElementById("canvas");
const offscreenCanvas = canvas.transferControlToOffscreen();

const worker = new Worker("worker.js");
worker.postMessage(
  {
    codec: ScrcpyVideoCodecId.H264,
    canvas: offscreenCanvas,
    stream: videoPacketStream,
  },
  [offscreenCanvas, videoPacketStream],
);
worker.addEventListener("message", (e) => {
  const { width, height } = e.data;
  canvas.width = width;
  canvas.height = height;
});
// worker.js
import type { ScrcpyVideoCodecId, ScrcpyMediaStreamPacket } from "@yume-chan/scrcpy";
import {
  WebGLWebCodecsDecoderRenderer,
  WebCodecsVideoDecoder,
} from "@yume-chan/scrcpy-decoder-webcodecs";

self.addEventListener("message", (e) => {
  const { codec, canvas, stream } = e.data as {
    codec: ScrcpyVideoCodecId;
    canvas: OffscreenCanvas;
    stream: ReadableStream<ScrcpyMediaStreamPacket>;
  };

  const renderer = new WebGLWebCodecsDecoderRenderer(canvas);
  const decoder = new WebCodecsVideoDecoder({ codec, renderer });

  decoder.sizeChanged(({ width, height }) => {
    postMessage({ width, height });
  });

  void stream.pipeTo(decoder.writable).catch((e) => {
    console.error(e);
  });
});

Take a screenshot​

Because WebCodecs decoder can render to different types of targets, it's more difficult to manually capture the latest frame.

To help with that, the decoder provides the snapshot method to easily capture the last rendered frame as a PNG image.

const blob = await decoder.snapshot();

Only when no frames has been rendered, the return value will be undefined.

Microsoft Edge on Windows​

By default, Chromium browsers uses FFMpeg internally for WebCodecs API. However, Microsoft Edge, when running on Windows, uses Media Foundation decoders instead.

Decoding H.265 requires the HEVC Video Extensions ($0.99) or HEVC Video Extensions from Device Manufacturer (free but not available anymore) app from Microsoft Store.

Decoding AV1 requires the AV1 Video Extension (free) app from Microsoft Store.