react-three-mind

react-three-mind

React Components adding Augmented Reality capabilities to @react-three/fiber, thanks to MindAR.

npm i react-three-mind

📍 Motivation

There's no easy and ready-to-use way of developing AR experiences whilst leveraging on the amazing ecosystem around @react-three/fiber. MindAR is a performance-oriented and easy to use library managing image and face tracking, but only supports AFrame or vanilla Three.js. This library aims to bridge the two worlds, while waiting for the new WebXR Standard to include image and face tracking.

📚 Usage

Provide an imageTargets url to toggle image tracking mode. See the examples and the original MindAR Documentation to find out how to compile your own image targets.

import React from "react";
import { createRoot } from "react-dom/client";

import { ARView, ARFaceMesh } from "react-three-mind";

const container = document.getElementById("root");
const root = createRoot(container);
root.render(
  <ARView>
    <ARFaceMesh>
      <meshBasicMaterial color="hotpink" wireframe />
    </ARFaceMesh>
  </ARView>
);

👩‍💻 API

ARView

AR Wrapper over @react-three/fiber Canvas managing the live video background and the 3D scene alignment.

const ref = useRef();
const startTracking = ref.current.startTracking(); // Starts tracking
const stopTracking = ref.current.stopTracking(); // Stops tracking
const switchCamera = ref.current.switchCamera(); // Switches between environment and user camera

<ARView
  ref={ref}
  autoplay // Automatically starts tracking once the camera stream is ready
  flipUserCamera={false} // Prevents automatic flipping of the user camera
  imageTargets={`url`} // URL of the generated image targets features
  maxTrack={1} // Maximum number of targets tracked simultaneously
  filterMinCF={0.1} // Cutoff Frequency, decrease to reduce jittering
  filterBeta={1000} // Speed Coefficient, increase to reduce delay
  warmupTolerance={5} // Number of continuous frames required for a target being detected to be marked as found
  missTolerance={5} // Number of continuous frames required for a target not being detected to be marked as lost
  {...canvasProps} // All @react-three/fiber Canvas props are valid
>
  <Scene />
</ARView>

ARAnchor

An Object3D anchor linking it to a tracked target. Can be used both for image and face targets.

<ARAnchor
  target={0} // Target (image or face) to be anchored to
  onAnchorFound={() => console.log(🥳)} // Callback invoked when anchor was found
  onAnchorLost={() => console.log(😢)} // Callback invoked when previously found anchor was lost
  {...groupProps} // All @react-three/fiber Group props are valid
>
  <mesh />
</ARAnchor>

ARFaceMesh

A Mesh Object representing a tracked face (see the original MindAR example).

<ARFaceMesh
  onFaceFound={() => console.log(🥳)} // Callback invoked when face was found
  onFaceLost={() => console.log(😢)} // Callback invoked when previously found face was lost
  {...meshProps} // All @react-three/fiber Mesh props are valid
>
  <meshBasicMaterial color="hotpink" wireframe />
</ARFaceMesh>

🙏 Credits

MindAR is developed by the amazing HiuKim Yuen. The showcase videos in this README come from its documentation.

Proudly supported by the amazing people @ ARuVR.

📮 TODO

• Add Showcase Video
• Fix Performance Issues
• Fix CI Build (Thanks @timothydang)