Skip to main content
Version: next

WebCodecs Decoder

Decode and render H.264, H.265 and AV1 streams in Web browsers using WebCodecs API, the new Web standard for hardware-accelerated video decoding.

It's fast, uses less hardware resources, and supports more profiles and levels. However, it's only available in recent versions of Chrome and Safari.

npm install @yume-chan/scrcpy-decoder-webcodecs

Feature detection​

To check if the WebCodecs API is available, you can check if VideoDecoder is defined:

const isSupported = globalThis.VideoDecoder !== undefined;

VideoDecoder.isConfigSupported is a static method that checks if a given codec configuration is supported.

It accepts a codec parameter string, for example "hev1.1.60.L153.B0.0.0.0.0.0" for H.265 and "av01.0.05M.08" for AV1.

const result = await VideoDecoder.isConfigSupported({
  codec: "hev1.1.60.L153.B0.0.0.0.0.0",
});
const isHevcSupported = result.supported === true;

You can decide which video codec to use based on the results.

Renderer​

WebCodecs API decodes video frames into VideoFrame objects. There are multiple methods to render those VideoFrame objects onto the page. This package provides three renderers:

info

These renderers are not tied to our WebCodecsVideoDecoder, they can also be used separately to render any VideoFrame objects from WebCodecs API.

  • InsertableStreamVideoFrameRenderer: Renders to a <video> element using Insertable Streams API. See quirks below.
  • WebGLVideoFrameRenderer: Renders to a <canvas> or OffscreenCanvas using WebGL. It only works with hardware accelerated WebGL, because without hardware acceleration, the performance is even worse than the bitmap renderer below.
  • BitmapVideoFrameRenderer: Renders to a <canvas> or OffscreenCanvas using bitmap renderer.
info

VideoFrames can also be rendered using 2D canvas. However, because it's slower than bitmap renderer, and bitmap renderer is already available on all devices, we didn't think it's necessary to implement it.

Quirks of Insertable Stream renderer​

The Insertable Streams renderer should be considered as experimental, because there are several issues around it:

Performance​

The Insertable Streams API is specifically designed to render video frames from WebCodecs API, but in reality it's only easier to integrate, not faster. So it doesn't have the performance advantage over other renderers.

Compatibility​

Its specification has two versions: the old MediaStreamTrackGenerator API, and the new VideoTrackGenerator. Only Chrome implemented the old API. The new API was added in mid 2023, but until end of 2024, nobody (including Chrome, who authored the specification), has implemented the new API (Chrome issue, Firefox issue).

As a result, we implemented the Insertable Stream renderer using the old MediaStreamTrackGenerator API. We will monitor the situation and update the renderer if necessary.

Lifecycle​

Because it renders to a <video> element, if the video element is removed from the DOM tree (e.g. to move it into another element, or another page), it will be automatically paused. You need to call renderer.element.play() to resume playback after adding it back to the DOM tree.

Create a renderer​

Generally, the performance ranking is InsertableStream ≈ WebGL >> Bitmap. However, because Insertable Stream renderer and WebGL renderer are not available on all devices, we recommend the following method to choose the best renderer on all devices:

InsertableStreamVideoFrameRenderer and WebGLVideoFrameRenderer both have an isSupported static property, to check whether they are supported by the current browser and hardware:

import type { VideoFrameRenderer } from "@yume-chan/scrcpy-decoder-webcodecs";
import {
  InsertableStreamVideoFrameRenderer,
  WebGLVideoFrameRenderer,
  BitmapVideoFrameRenderer,
} from "@yume-chan/scrcpy-decoder-webcodecs";

function createWebCodecsRenderer(): {
  renderer: VideoFrameRenderer;
  element: HTMLVideoElement | HTMLCanvasElement;
} {
  if (InsertableStreamVideoFrameRenderer.isSupported) {
    const renderer = new InsertableStreamVideoFrameRenderer();
    return { renderer, element: renderer.element };
  }

  if (WebGLVideoFrameRenderer.isSupported) {
    const renderer = new WebGLVideoFrameRenderer();
    return { renderer, element: renderer.canvas as HTMLCanvasElement };
  }

  const renderer = new BitmapVideoFrameRenderer();
  return { renderer, element: renderer.canvas as HTMLCanvasElement };
}

You will need to insert the created element into the page to display the video:

const { renderer, element } = createWebCodecsRenderer();
document.body.appendChild(element);

Or, all renderers accept existing rendering targets:

new InsertableStreamVideoFrameRenderer(videoElement);
new WebGLVideoFrameRenderer(canvasElementOrOffscreenCanvas);
new BitmapVideoFrameRenderer(canvasElementOrOffscreenCanvas);

Create a decoder​

import type { ScrcpyMediaStreamPacket } from "@yume-chan/scrcpy";
import { ScrcpyVideoCodecId } from "@yume-chan/scrcpy";
import type {
  ScrcpyVideoDecoder,
  ScrcpyVideoDecoderCapability,
} from "@yume-chan/scrcpy-decoder-tinyh264";
import { WritableStream } from "@yume-chan/stream-extra";
import type { VideoFrameRenderer } from "./render/index.js";

export declare class WebCodecsVideoDecoder implements ScrcpyVideoDecoder {
  static get isSupported(): boolean;
  static readonly capabilities: Record<string, ScrcpyVideoDecoderCapability>;

  get codec(): ScrcpyVideoCodecId;
  get writable(): WritableStream<ScrcpyMediaStreamPacket>;
  get renderer(): VideoFrameRenderer;

  get framesRendered(): number;
  get framesSkipped(): number;

  get sizeChanged(): import("@yume-chan/event").AddEventListener<
    {
      width: number;
      height: number;
    },
    unknown
  >;

  /**
   * Create a new WebCodecs video decoder.
   */
  constructor(options: WebCodecsVideoDecoder.Options);

  snapshot(): Promise<Blob | undefined>;
  dispose(): void;
}

export declare namespace WebCodecsVideoDecoder {
  interface Options {
    /**
     * The video codec to decode
     */
    codec: ScrcpyVideoCodecId;

    renderer: VideoFrameRenderer;
  }
}

The constructor requires two options:

  • codec: the video codec to be decoded. It can be retrieved from the video stream metadata, or hard-coded if you only use a specific video codec.
  • renderer: a renderer created in the previous section.

Similar to the TinyH264 decoder, after creating a decoder instance, you need to pipe the video stream into the writable stream:

const decoder = new WebCodecsVideoDecoder({
  codec: videoMetadata.codec,
  renderer: renderer,
});

void videoPacketStream.pipeTo(decoder.writable).catch((e) => {
  console.error(e);
});

Take a screenshot​

Because WebCodecs decoder can render to different types of targets, it's more difficult to manually capture the latest frame.

To help with that, the decoder provides the snapshot method to easily capture the last rendered frame as a PNG image.

const blob = await decoder.snapshot();

Only when no frames has been rendered, the return value will be undefined.

Microsoft Edge on Windows​

By default, Chromium browsers uses FFMpeg internally for WebCodecs API. However, Microsoft Edge, when running on Windows, uses Media Foundation decoders instead.

Decoding H.265 requires the HEVC Video Extensions ($0.99) or HEVC Video Extensions from Device Manufacturer (free but not available anymore) app from Microsoft Store.

Decoding AV1 requires the AV1 Video Extension (free) app from Microsoft Store.