node-av

NodeAV

Native Node.js bindings for FFmpeg with full TypeScript support - including type-safe options (autocomplete + validation) for every codec, format, filter, and bitstream filter, generated from FFmpeg's own metadata. Provides direct access to FFmpeg's C APIs through N-API. Includes both raw FFmpeg bindings for full control and higher-level abstractions. Automatic resource management via Disposable pattern, hardware acceleration support and prebuilt binaries for Windows, Linux, and macOS.

📚 Documentation

Table of Contents

• Installation
• Quick Start
  • Low-Level API
  • High-Level API
  • Pipeline API
• Hardware Acceleration
  • Auto-Detection
  • Specific Hardware
• Stream Processing
  • From Files or Network
  • From Buffers
  • Raw Media Processing
• Device Capture
• FFmpeg Binary Access
• Resource Management
• Imports and Tree Shaking
• Key Features
• Performance
  • Benchmarks
  • Sync vs Async Operations
• Memory Safety Considerations
• Electron
• Examples
• Prebuilt Binaries
• Troubleshooting
• License
• Contributing
• Support
• See Also

Installation

npm install node-av

Quick Start

Low-Level API

Direct access to FFmpeg's C APIs with minimal abstractions. Perfect when you need full control over FFmpeg functionality.

import { AVERROR_EOF, AVMEDIA_TYPE_VIDEO } from 'node-av/constants';
import { Codec, CodecContext, FFmpegError, FormatContext, Frame, Packet, Rational } from 'node-av/lib';

// Open input file
await using ifmtCtx = new FormatContext();

let ret = await ifmtCtx.openInput('input.mp4');
FFmpegError.throwIfError(ret, 'Could not open input file');

ret = await ifmtCtx.findStreamInfo();
FFmpegError.throwIfError(ret, 'Could not find stream info');

// Find video stream
const videoStreamIndex = ifmtCtx.findBestStream(AVMEDIA_TYPE_VIDEO);
const videoStream = ifmtCtx.streams?.[videoStreamIndex];

if (!videoStream) {
  throw new Error('No video stream found');
}

// Create codec
const codec = Codec.findDecoder(videoStream.codecpar.codecId);
if (!codec) {
  throw new Error('Codec not found');
}

// Allocate codec context for the decoder
using decoderCtx = new CodecContext();
decoderCtx.allocContext3(codec);

ret = decoderCtx.parametersToContext(videoStream.codecpar);
FFmpegError.throwIfError(ret, 'Could not copy codec parameters to decoder context');

// Open decoder context
ret = await decoderCtx.open2(codec, null);
FFmpegError.throwIfError(ret, 'Could not open codec');

// Process packets
using packet = new Packet();
packet.alloc();

using frame = new Frame();
frame.alloc();

while (true) {
  let ret = await ifmtCtx.readFrame(packet);
  if (ret < 0) {
    break;
  }

  if (packet.streamIndex === videoStreamIndex) {
    // Send packet to decoder
    ret = await decoderCtx.sendPacket(packet);
    if (ret < 0 && ret !== AVERROR_EOF) {
      FFmpegError.throwIfError(ret, 'Error sending packet to decoder');
    }

    // Receive decoded frames
    while (true) {
      const ret = await decoderCtx.receiveFrame(frame);
      if (ret === AVERROR_EOF || ret < 0) {
        break;
      }

      console.log(`Decoded frame ${frame.pts}, size: ${frame.width}x${frame.height}`);

      // Process frame data...
    }
  }

  packet.unref();
}

High-Level API

Higher-level abstractions for common tasks like decoding, encoding, filtering, and transcoding. Easier to use while still providing access to low-level details when needed.

import { Decoder, Demuxer, Encoder, HardwareContext, Muxer } from 'node-av/api';
import { FF_ENCODER_LIBX264 } from 'node-av/constants';

// Open Demuxer
await using input = await Demuxer.open('input.mp4');

// Get video stream
const videoStream = input.video()!;

// Optional, setup hardware acceleration
using hw = HardwareContext.auto();

// Create decoder
using decoder = await Decoder.create(videoStream, {
  hardware: hw, // Optional, use hardware acceleration if available
});

// Create encoder
using encoder = await Encoder.create(FF_ENCODER_LIBX264, {
  decoder, // Optional, copy settings from decoder
});

// Open Muxer
await using output = await Muxer.open('output.mp4', {
  input, // Optional, used to copy global headers and metadata
});

// Add stream to output
const outputIndex = output.addStream(encoder, {
  inputStream: videoStream, // Optional, copy settings from input stream
});

// Create processing generators
const inputGenerator = input.packets(videoStream.index);
const decoderGenerator = decoder.frames(inputGenerator);
const encoderGenerator = encoder.packets(decoderGenerator);

// Process packets
for await (using packet of encoderGenerator) {
  await output.writePacket(packet, outputIndex);
}

// Done

Pipeline API

A simple way to chain together multiple processing steps like decoding, filtering, encoding, and muxing.

import { Decoder, Demuxer, Encoder, HardwareContext, Muxer, pipeline } from 'node-av/api';
import { FF_ENCODER_LIBX264 } from 'node-av/constants';

// Simple transcode pipeline: input → decoder → encoder → output

// Open Demuxer
await using input = await Demuxer.open('input.mp4');

// Get video stream
const videoStream = input.video()!;

// Optional, setup hardware acceleration
using hw = HardwareContext.auto();

// Create decoder
using decoder = await Decoder.create(videoStream, {
  hardware: hw, // Optional, use hardware acceleration if available
});

// Create encoder
using encoder = await Encoder.create(FF_ENCODER_LIBX264, {
  decoder, // Optional, copy settings from decoder
});

// Open Muxer
await using output = await Muxer.open('output.mp4', {
  input, // Optional, used to copy global headers and metadata
});

const control = pipeline(input, decoder, encoder, output);
await control.completion;

Hardware Acceleration

The library supports all hardware acceleration methods available in FFmpeg. The specific hardware types available depend on your FFmpeg build and system configuration.

Auto-Detection

import { HardwareContext } from 'node-av/api';
import { FF_ENCODER_LIBX264 } from 'node-av/constants';

// Automatically detect best available hardware
const hw = HardwareContext.auto();
console.log(`Using hardware: ${hw.deviceTypeName}`);

// Use with decoder
const decoder = await Decoder.create(stream, {
  hardware: hw
});

// Use with encoder (use hardware-specific codec)
const encoderCodec = hw?.getEncoderCodec('h264') ?? FF_ENCODER_LIBX264;
const encoder = await Encoder.create(encoderCodec, {
  decoder,
});

Specific Hardware

import { AV_HWDEVICE_TYPE_CUDA, AV_HWDEVICE_TYPE_VAAPI } from 'node-av/constants';

// Use specific hardware type
const cuda = HardwareContext.create(AV_HWDEVICE_TYPE_CUDA);
const vaapi = HardwareContext.create(AV_HWDEVICE_TYPE_VAAPI, '/dev/dri/renderD128');

Stream Processing

From Files or Network

const media = await Demuxer.open('input.mp4');

// or

const media = await Demuxer.open('rtsp://example.com/stream');

From Buffers

import { readFile } from 'fs/promises';

const buffer = await readFile('input.mp4');
const media = await Demuxer.open(buffer);

Custom I/O Callbacks

import type { IOInputCallbacks, IOOutputCallbacks } from 'node-av/api';

// Custom input source
const inputCallbacks: IOInputCallbacks = {
  read: (size: number) => {
    // Read from your custom source
    return buffer; // or null for EOF
  },
  seek: (offset: bigint, whence: number) => {
    // Seek in your custom source
    return newPosition;
  }
};

await using input = await Demuxer.open(inputCallbacks, {
  format: 'mp4'
});

Raw Media Processing

// Raw video input
const rawVideo = await Demuxer.open({
  type: 'video',
  input: 'input.yuv',
  width: 1280,
  height: 720,
  pixelFormat: AV_PIX_FMT_YUV420P,
  frameRate: { num: 30, den: 1 }
});

// Raw audio input
const rawAudio = await Demuxer.open({
  type: 'audio',
  input: 'input.pcm',
  sampleRate: 48000,
  channels: 2,
  sampleFormat: AV_SAMPLE_FMT_S16
}, {
  format: 's16le'
});

Device Capture

Capture video, audio, and screen content directly from system devices. Platform-specific formats are handled automatically.

import { DeviceAPI } from 'node-av/api';

// List devices and query capabilities
const devices = await DeviceAPI.list();
const modes = await DeviceAPI.modes(devices.find(d => d.type === 'video')!.name);

// Camera
await using camera = await DeviceAPI.openCamera({ videoDevice: 0, width: 1280, height: 720, frameRate: 30 });

// Microphone
await using mic = await DeviceAPI.openMicrophone({ audioDevice: 0, sampleRate: 48000, channels: 2 });

// Combined video + audio (macOS/Windows)
await using device = await DeviceAPI.openDevice({ videoDevice: 0, audioDevice: 0, width: 1280, height: 720, frameRate: 30 });

// Screen capture
await using screen = await DeviceAPI.openScreen({ frameRate: 30, drawMouse: true });

// Screen capture with system audio (macOS 13.0+)
await using screen2 = await DeviceAPI.openScreen({
  frameRate: 30,
  avfoundation: { captureSystemAudio: true, audioSampleRate: 48000, audioChannels: 2 },
});

Platform	Video	Audio	Screen
macOS	AVFoundation	AVFoundation	AVFoundation (ScreenCaptureKit)
Linux	V4L2	ALSA	x11grab
Windows	DirectShow	DirectShow	GDIGrab

FFmpeg Binary Access

Need direct access to the FFmpeg binary? The library provides an easy way to get FFmpeg binaries that automatically downloads and manages platform-specific builds.

import { ffmpegPath, isFfmpegAvailable } from 'node-av/ffmpeg';
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';

const execFileAsync = promisify(execFile);

// Check if FFmpeg binary is available
if (isFfmpegAvailable()) {
  console.log('FFmpeg binary found at:', ffmpegPath());

  // Use FFmpeg binary directly
  const { stdout } = await execFileAsync(ffmpegPath(), ['-version']);
  console.log(stdout);
} else {
  console.log('FFmpeg binary not available - install may have failed');
}

// Direct usage example
async function convertVideo(input: string, output: string) {
  const args = [
    '-i', input,
    '-c:v', 'libx264',
    '-crf', '23',
    '-c:a', 'aac',
    output
  ];

  await execFileAsync(ffmpegPath(), args);
}

The FFmpeg binary is automatically downloaded during installation from GitHub releases and matches the same build used by the native bindings.

Resource Management

The library supports automatic resource cleanup using the Disposable pattern:

// Automatic cleanup with 'using'
{
  await using media = await Demuxer.open('input.mp4');
  using decoder = await Decoder.create(media.video());
  // Resources automatically cleaned up at end of scope
}

// Manual cleanup
const media = await Demuxer.open('input.mp4');
try {
  // Process media
} finally {
  await media.close();
}

Imports and Tree Shaking

The library provides multiple entry points for optimal tree shaking:

// High-Level API only - Recommended for most use cases
import { Demuxer, Muxer, Decoder, Encoder } from 'node-av/api';

// Low-Level API only - Direct FFmpeg bindings
import { FormatContext, CodecContext, Frame, Packet } from 'node-av/lib';

// Constants only - When you just need FFmpeg constants
import { AV_PIX_FMT_YUV420P, AV_CODEC_ID_H264 } from 'node-av/constants';

// Channel layouts only - For audio channel configurations
import { AV_CHANNEL_LAYOUT_STEREO, AV_CHANNEL_LAYOUT_5POINT1 } from 'node-av/layouts';

// Default export - Includes everything
import * as ffmpeg from 'node-av';

Key Features

Beyond basic transcoding, NodeAV provides advanced media processing capabilities:

Fully Typed FFmpeg Options Every FFmpeg option is typed - across codecs, formats, ~580 filters, and bitstream filters. Keys autocomplete, enum values are validated, and typos become compile-time errors, all generated directly from FFmpeg's AVOption metadata (including each option's description and a link to the FFmpeg docs as JSDoc). No more grepping the FFmpeg docs for flag names.

Speech Recognition with Whisper Integrate automatic speech-to-text transcription using OpenAI's Whisper model through the whisper.cpp implementation. The library handles automatic model downloading from HuggingFace, supports multiple model sizes (tiny, base, small, medium, large) for different accuracy/performance tradeoffs, and provides hardware-accelerated inference through Metal (macOS), Vulkan (cross-platform), or OpenCL backends. Transcription results include precise timestamps and can be processed in real-time from any audio source.

Advanced Video Filtering with FilterComplexAPI Build sophisticated video processing pipelines using FFmpeg's complete filter ecosystem. The FilterComplexAPI provides direct access to complex filtergraphs with multiple inputs and outputs, enabling advanced operations like picture-in-picture overlays, multi-stream composition (side-by-side, grid layouts), real-time video effects, and custom processing chains. All filters support hardware acceleration where available, and filter configurations can be dynamically constructed based on runtime requirements.

Browser Streaming Stream any media source directly to web browsers through fragmented MP4 (fMP4) or WebRTC protocols. The library can process inputs from RTSP cameras, local files, network streams, or custom sources and package them for browser consumption with minimal latency. Complete examples demonstrate both Media Source Extensions (MSE) based playback for on-demand content and WebRTC integration for real-time streaming scenarios.

RTSP Backchannel / Talkback Implements bidirectional RTSP communication for IP camera integration. The library provides native support for RTSP backchannel streams, enabling audio transmission to camera devices. Transport is handled automatically with support for both TCP (interleaved mode) and UDP protocols, with proper RTP packet formatting and stream synchronization.

Image Scaling & Snapshots The Scaler turns decoded frames into raw pixel buffers (rgb/rgba/gray/nv12/yuv420p) or JPEG/PNG images for detection, thumbnail, and snapshot workloads. One reusable instance pools its swscale contexts, GPU filter graphs, and encoders for zero per-frame allocation; hardware frames are cropped/scaled on the GPU with only the small result downloaded. Recurring resolutions can also be served directly by EncoderPool.

See the Examples section for complete implementations.

Performance

NodeAV executes all media operations directly through FFmpeg's native C libraries. The Node.js bindings add minimal overhead - mostly just the JavaScript-to-C boundary crossings. During typical operations like transcoding or filtering, most processing time is spent in FFmpeg's optimized C code.

Benchmarks

Performance comparison with FFmpeg CLI (4K 60fps, 30s test files on Apple M3 Max):

Operation	FFmpeg CLI (FPS)	node-av (FPS)	FFmpeg CLI (Time)	node-av (Time)	Diff
SW H.264 Transcode	96 fps	96 fps	18.7s	18.7s	≈0%
SW H.265 Transcode	40 fps	41 fps	44.5s	43.7s	+1.5%
HW H.264 Transcode	55 fps	55 fps	33.0s	32.8s	+0.5%
Stream Copy (Remux)	48k fps	31k fps	38ms	106ms	-35%

Memory Usage:

Operation	FFmpeg CLI	node-av	Difference
H.264 Transcode (4K)	3.6 GB	3.4 GB	-5%
Stream Copy	28 MB	1 MB	-96%

📊 Full benchmark results

Sync vs Async Operations

Every async method in NodeAV has a corresponding synchronous variant with the Sync suffix:

• Async methods (default) - Non-blocking operations using N-API's AsyncWorker. Methods like decode(), encode(), read(), packets() return Promises or AsyncGenerators.

• Sync methods - Direct FFmpeg calls without AsyncWorker overhead. Same methods with Sync suffix: decodeSync(), encodeSync(), readSync(), packetsSync().

The key difference: Async methods don't block the Node.js event loop, allowing other operations to run concurrently. Sync methods block until completion but avoid AsyncWorker overhead, making them faster for sequential processing.

Memory Safety Considerations

NodeAV provides direct bindings to FFmpeg's C APIs, which work with raw memory pointers. The high-level API adds safety abstractions and automatic resource management, but incorrect usage can still cause crashes. Common issues include mismatched video dimensions, incompatible pixel formats, or improper frame buffer handling. The library validates parameters where possible, but can't guarantee complete memory safety without limiting functionality. When using the low-level API, pay attention to parameter consistency, resource cleanup, and format compatibility. Following the documented patterns helps avoid memory-related issues.

Electron

NodeAV fully supports Electron applications. The prebuilt binaries are ABI-compatible with Electron, so no native rebuild is required during packaging. Both the native bindings and the bundled FFmpeg CLI binaries work seamlessly within Electron's main process.

NodeAV also supports Electron's offscreen rendering with shared textures, enabling zero-copy GPU texture import for hardware-accelerated encoding. See the Electron Builder and Electron Forge examples for complete implementations.

If you encounter module resolution errors like Cannot find module 'lib/binary-stream', add this override to your project's package.json:

{
  "overrides": {
    "@shinyoshiaki/binary-data": "npm:@seydx/binary-data@0.6.2"
  }
}

Examples

Example	FFmpeg	Low-Level API	High-Level API
browser-fmp4			✓
browser-webrtc			✓
electron-builder			✓
electron-forge			✓
api-abort-signal			✓
api-bitstream-filter			✓
api-dash			✓
api-decoder-resample			✓
api-device-capture			✓
api-encode-decode			✓
api-filter-complex			✓
api-filter-complex-grid			✓
api-filter-complex-typed			✓
api-fmp4			✓
api-frame-extract			✓
api-hw-codecs			✓
api-hw-decode-sw-encode			✓
api-hw-raw			✓
api-hw-raw-output			✓
api-hw-rtsp			✓
api-hw-stream-custom-io			✓
api-hw-transcode			✓
api-hw-filter-sync			✓
api-muxing			✓
api-pipeline-hw-rtsp			✓
api-pipeline-progress			✓
api-pipeline-raw-muxing			✓
api-probe			✓
api-rtp			✓
api-rtsp-backchannel			✓
api-scaler			✓
api-sdp-custom			✓
api-sdp-input			✓
api-snapshot			✓
api-stream-input			✓
api-sw-decode-hw-encode			✓
api-sw-transcode			✓
api-whisper-subtitles			✓
api-whisper-transcribe			✓
frame-utils		✓
avio-read-callback	✓	✓
avio-async-read-callback		✓
decode-audio	✓	✓
decode-filter-audio	✓	✓
decode-filter-video	✓	✓
decode-video	✓	✓
demux-decode	✓	✓
encode-audio	✓	✓
encode-video	✓	✓
ffprobe-metadata		✓
filter-audio	✓	✓
hw-decode	✓	✓
hw-encode		✓
hw-transcode		✓
qsv-decode	✓
qsv-transcode	✓
vaapi-encode	✓
vaapi-transcode	✓
mux	✓	✓
remux	✓	✓
resample-audio	✓	✓
rtsp-stream-info		✓
scale-video	✓	✓
show-metadata	✓	✓
transcode-aac	✓	✓
transcode	✓	✓

Prebuilt Binaries

Prebuilt binaries are available for multiple platforms:

• macOS: x64, ARM64
• Linux: x64, ARM64
• Windows: x64, ARM64 (automatic MSVC/MinGW selection)

Troubleshooting

Hardware Acceleration on Linux (Intel/VAAPI)

For hardware-accelerated video processing with Intel GPUs on Linux, you need to install specific system packages. The FFmpeg binaries included with this library are built with libva 2.20, which requires Ubuntu 24.04+ or Debian 13+ as minimum OS versions.

Installation Steps

• Add Kisak-Mesa PPA (recommended for newer Mesa versions with better hardware support):

sudo add-apt-repository ppa:kisak/kisak-mesa
sudo apt update

• Install required packages:

sudo apt install libmfx-gen1.2 mesa-va-drivers mesa-vulkan-drivers libva2 libva-drm2 vainfo libvulkan1 vulkan-tools

After installation, verify hardware acceleration is working:

# Check VAAPI support
vainfo

# Check Vulkan support
vulkaninfo

# Should show available profiles and entrypoints for your Intel GPU

Note: If you're running an older Ubuntu version (< 24.04) or Debian version (< 13), you'll need to upgrade your OS to use hardware acceleration with this library.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Important: FFmpeg itself is licensed under LGPL/GPL. Please ensure compliance with FFmpeg's license terms when using this library. The FFmpeg libraries themselves retain their original licenses, and this wrapper library does not change those terms. See FFmpeg License for details.

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for development setup, code standards, and contribution guidelines before submitting pull requests.

Support

For issues and questions, please use the GitHub issue tracker.

See Also

• FFmpeg Documentation
• FFmpeg Doxygen
• Jellyfin FFmpeg
• FFmpeg MSVC

Star History