vision-camera-resize-plugin

vision-camera-resize-plugin

A VisionCamera Frame Processor Plugin for fast and efficient Frame resizing, cropping and pixel-format conversion (YUV -> RGB) using GPU-acceleration and CPU-vector based operations.

Installation

1. Install react-native-vision-camera and make sure Frame Processors are enabled.
2. Install vision-camera-resize-plugin:

   yarn add vision-camera-resize-plugin
   cd ios && pod install
   

Usage

Use the resize plugin within a Frame Processor:

const { resize } = useResizePlugin()

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'

  const resized = resize(frame, {
    size: {
      width: 192,
      height: 192
    },
    pixelFormat: 'rgb-uint8'
  })
  const array = new Uint8Array(resized)

  const firstPixel = {
    r: array[0],
    g: array[1],
    b: array[2]
  }
}, [])

Or outside of a function component:

const { resize } = createResizePlugin()

const frameProcessor = createFrameProcessor((frame) => {
  'worklet'

  const resized = resize(frame, {
    // ...
  })
  // ...
})

Pixel Formats

The resize plugin operates in RGB colorspace, and all values are in uint8.

Name	0	1	2	3
rgb-uint8	R	G	B	R
rgba-uint8	R	G	B	A
argb-uint8	A	R	G	B
bgra-uint8	B	G	R	A
bgr-uint8	B	G	R	B
abgr-uint8	A	B	G	R

Performance

If possible, use one of these two formats:

• argb-uint8: Can be converted the fastest, but has an additional unused alpha channel.
• rgb-uint8: Requires one more conversion step from argb-uint8, but uses 25% less memory due to the removed alpha channel.

All other formats require additional conversion steps, and float models have additional memory overhead (up to 4x as big).

When using TensorFlow Lite, try to convert your model to use argb-uint8 or rgb-uint8 as it's input type.

react-native-fast-tflite

The vision-camera-resize-plugin can be used together with react-native-fast-tflite to prepare the input tensor data.

For example, to use the efficientdet TFLite model to detect objects inside a Frame, simply add the model to your app's bundle, set up VisionCamera and react-native-fast-tflite, and resize your Frames accordingly.

From the model's description on the website, we understand that the model expects 320 x 320 x 3 buffers as input, where the format is uint8 rgb.

const objectDetection = useTensorflowModel(require('assets/efficientdet.tflite'))
const model = objectDetection.state === "loaded" ? objectDetection.model : undefined

const { resize } = useResizePlugin()

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'

  const data = resize(frame, {
    size: {
      width: 320,
      height: 320,
    },
    pixelFormat: 'rgb-uint8'
  })
  const output = model.runSync([data])

  const numDetections = output[0]
  console.log(`Detected ${numDetections} objects!`)
}, [model])

Benchmarks

I benchmarked vision-camera-resize-plugin on an iPhone 15 Pro, using the following code:

const start = performance.now()
const result = resize(frame, {
  size: {
    width: 100,
    height: 100,
  },
  pixelFormat: 'rgb-uint8',
})
const end = performance.now()

const diff = (end - start).toFixed(2)
console.log(`Resize and conversion took ${diff}ms!`)

And when running on 1080x1920 yuv Frames, I got the following results:

 LOG  Resize and conversion took 6.48ms
 LOG  Resize and conversion took 6.06ms
 LOG  Resize and conversion took 5.89ms
 LOG  Resize and conversion took 5.97ms
 LOG  Resize and conversion took 6.98ms

This means the Frame Processor can run at up to ~160 FPS.

Adopting at scale

This library is provided as is, I work on it in my free time.

If you're integrating vision-camera-resize-plugin in a production app, consider funding this project and contact me to receive premium enterprise support, help with issues, prioritize bugfixes, request features, help at integrating vision-camera-resize-plugin and/or VisionCamera Frame Processors, and more.

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

License

MIT

---

Made with create-react-native-library