FFMediaToolkit

FFMediaToolkit

FFMediaToolkit is a .NET library for creating and reading multimedia files. It uses native FFmpeg libraries by the FFmpeg.Autogen bindings.

Features

• Decoding/encoding audio-video files in many formats supported by FFmpeg.
• Extracting audio data as floating point arrays.
• Access to any video frame by timestamp.
• Creating videos from images with metadata, pixel format, bitrate, CRF, FPS, GoP, dimensions and other codec settings.
• Supports reading multimedia chapters and metadata.

Code samples

• Extract all video frames as PNG files

  // Open video file
  using var file = MediaFile.Open(@"D:\example\movie.mp4", new MediaOptions() { VideoPixelFormat = ImagePixelFormat.Rgba32 });
  
  // Get pixel buffer from SkiaSharp bitmap
  using var bitmap = new SKBitmap(file.Video.Info.FrameSize.Width, file.Video.Info.FrameSize.Height, SKColorType.Rgba8888, SKAlphaType.Unpremul);
  var pixelBuffer = bitmap.GetPixelSpan();
  
  int i = 0;
  // Iterate over all frames in the video - decoded frame will be written to the buffer
  while (file.Video.TryGetNextFrame(pixelBuffer))
  {
      // Save image as PNG file
      using var fs = File.OpenWrite($@"D:\example\frame_{i++}.png");
      bitmap.Encode(fs, SKEncodedImageFormat.Png, 100);
  }
  

  This example uses SkiaSharp to save decoded frames. See Usage details section for examples with other graphics libraries.

• Video decoding

  // Open a multimedia file
  // You can use the MediaOptions properties to set decoder options
  var file = MediaFile.Open(@"C:\videos\movie.mp4");
  
  // Get the frame at 5th second of the video
  var frame = file.Video.GetFrame(TimeSpan.FromSeconds(5));
  // ...
  // Frame should be disposed when no longer needed
  frame.Dispose(); 
  
  // Print information about the video stream
  Console.WriteLine($"Bitrate: {file.Info.Bitrate / 1000.0} kb/s");
  var info = file.Video.Info;
  Console.WriteLine($"Duration: {info.Duration}");
  Console.WriteLine($"Frames count: {info.NumberOfFrames ?? "N/A"}");
  var frameRateInfo = info.IsVariableFrameRate ? "average" : "constant";
  Console.WriteLine($"Frame rate: {info.AvgFrameRate} fps ({frameRateInfo})");
  Console.WriteLine($"Frame size: {info.FrameSize}");
  Console.WriteLine($"Pixel format: {info.PixelFormat}");
  Console.WriteLine($"Codec: {info.CodecName}");
  Console.WriteLine($"Is interlaced: {info.IsInterlaced}");
  

• Encode video from images.

  // You can set codec, bitrate, frame rate and many other options here
  var settings = new VideoEncoderSettings(width: 1920, height: 1080, framerate: 30, codec: VideoCodec.H264) {
      EncoderPreset = EncoderPreset.Fast,
      CRF = 17,
  };
  // Create output file
  using var file = MediaBuilder.CreateContainer(@"D:\example\video.mp4").WithVideo(settings).Create();
  for(int i = 0; i < 300; i++)
  {
      // Load image using SkiaSharp (other libraries are also supported if provide access to pixel buffer)
      using var bmp = SKBitmap.Decode($@"D:\example\frame_{i}.png");
      // Encode the video frame
      file.Video.AddFrame(new ImageData(bmp.GetPixelSpan(), ImagePixelFormat.Rgba32, bmp.Width, bmp.Height));
  }
  

Setup

Install the FFMediaToolkit package from NuGet.

dotnet add package FFMediaToolkit

FFmpeg libraries are not included in the package. To use FFMediaToolkit, you need the FFmpeg shared build binaries: avcodec (v61), avformat (v61), avutil (v59), swresample (v5), swscale (v8).

Supported FFmpeg version: 7.x (shared build)

• Windows - You can download it from the BtbN/FFmpeg-Builds or gyan.dev. You only need *.dll files from the .\bin directory (not .\lib) of the ZIP package. Place the binaries in .\runtimes\win-[x64\arm64]\native\ in the application output directory or set FFmpegLoader.FFmpegPath.
• Linux - Download FFmpeg using your package manager. Default path is /usr/lib/*-linux-gnu
• macOS, iOS, Android - Not supported.

You need to set FFmpegLoader.FFmpegPath with a full path to FFmpeg libraries.

In .NET Framework projects you have to disable the Build -> Prefer 32-bit option in Visual Studio project properties.

Usage details

FFMediaToolkit supports decoding video frames into pixel buffers which can be Span<byte>, byte[] or unmanaged memory. You can specify target pixel format by setting the MediaOptions.VideoPixelFormat property. The default format is Bgr24.

If you want to process or save the decoded frame, you can pass it to another graphics library, as shown in the examples below.

• For SkiaSharp library:

  • Video decoding

    using var file = MediaFile.Open(@"D:\example\video.mp4", new MediaOptions() {
        StreamsToLoad = MediaMode.Video, 
        VideoPixelFormat = ImagePixelFormat.Rgba32
    });
    using var bitmap = new SKBitmap(file.Video.Info.FrameSize.Width, file.Video.Info.FrameSize.Height, SKColorType.Rgba8888, SKAlphaType.Unpremul);
    var buffer = bitmap.GetPixelSpan();
    
    while(file.Video.TryGetNextFrame(buffer)) {
        // do something
    }
    

  • Video encoding

    using var bmp = SKBitmap.Decode($@"D:\example\frame.png");
    mediaFile.Video.AddFrame(new ImageData(bmp.GetPixelSpan(), ImagePixelFormat.Rgba32, bmp.Width, bmp.Height));
    

• For ImageSharp library:

  • Video decoding

    var buffer = new byte[file.Video.FrameByteCount];
    var bmp = Image.WrapMemory<Bgr24>(buffer, file.Video.Info.FrameSize.Width, file.Video.Info.FrameSize.Height);
    
    while(file.Video.TryGetNextFrame(buffer)) {
        // do something
    }
    

• For GDI+ System.Drawing.Bitmap (Windows only):

  • Video decoding

    // Create bitmap once
    var rect = new Rectangle(Point.Empty, file.Video.Info.FrameSize);
    var bitmap = new Bitmap(rect.Width, rect.Height, PixelFormat.Format24bppRgb);
    // ...
    // Read next frame
    var bitLock = bitmap.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format24bppRgb);
    file.Video.TryGetNextFrame(bitLock.Scan0, bitLock.Stride);
    bitmap.UnlockBits(bitLock);
    

  • Video encoding

    var rect = new Rectangle(Point.Empty, bitmap.Size);
    var bitLock = bitmap.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb);
    
    var bitmapData = ImageData.FromPointer(bitLock.Scan0, ImagePixelFormat.Bgr24, bitmap.Size);
    mediaFile.Video.AddFrame(bitmapData); // Encode the frame
    
    bitmap.UnlockBits(bitLock); // UnlockBits() must be called after encoding the frame
    

• For desktop apps with WPF UI (Windows only):

  • Video decoding

    using System.Windows.Media.Imaging;
    
    // Create bitmap once
    var bmp = new WriteableBitmap(media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height, 96, 96, PixelFormats.Bgr24, null);
    // ...
    // Read next frame
    bmp.Lock();
    var success = media.Video.TryGetNextFrame(bmp.BackBuffer, bmp.BackBufferStride);
    if(success) {
        bmp.AddDirtyRect(new Int32Rect(0, 0, media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height));
    }
    bmp.Unlock();
    

  • Video encoding

    var bitmapSource = new BitmapImage(new Uri(@"D:\example\image.png"));
    var wb = new WriteableBitmap(bitmap);
    mediaFile.Video.AddFrame(ImageData.FromPointer(wb.BackBuffer, ImagePixelFormat.Bgra32, wb.PixelWidth, wb.PixelHeight));
    

Visual Basic usage

Writing decoded bitmap directly to the WPF WriteableBitmap buffer:

Dim file As FileStream = New FileStream("path to the video file", FileMode.Open, FileAccess.Read)
Dim media As MediaFile = MediaFile.Load(file)
Dim bmp As WriteableBimap = New WriteableBitmap(media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height, 96, 96, PixelFormats.Bgr24, Nothing)
bmp.Lock()
Dim decoded As Boolean = media.Video.TryGetFrame(TimeSpan.FromMinutes(1), bmp.BackBuffer, bmp.BackBufferStride)
If decoded Then
    bmp.AddDirtyRect(New Int32Rect(0, 0, media.Video.Info.FrameSize.Width, media.Video.Info.FrameSize.Height))
End If
bmp.Unlock()
imageBox.Source = bmp

Converting ImageData to a byte array:

Dim data() As Byte = media.Video.GetNextFrame().Data.ToArray()

Licensing

This project is licensed under the MIT license.