react-mindee-js

react-mindee-js

Computer vision SDK for image segmentation and document processing on top of mindee APIs

Check out the storybook.

Features

This SDK was made for building interfaces on top of Mindee document parsing APIs and more generally on top of any computer vision detection APIs.

• Work with images and pdfs
• Display interactive shapes on images or pdfs with custom style
• Bind custom functions to shapes mouse events
• Create multi-pages pdfs navigation sidebar
• Zoom in and out in the canvas
• Move the image with the mouse
• Create a lens to zoom around the user's pointer

This SDK was primarily made for document processing but can be used for any type of computer vision interface:

Compatibility

The React SDK is compatible with React 16.8.0 +

Contents

• Installation
• Use cases
  • Hello world
  • Handle multi-pages pdfs navigation
  • Add lens
  • Mindee APIs helpers
• Components
  • <AnnotationViewer />
  • <AnnotationLens />
  • <AnnotationSidebar />
  • <Fullscreen />
• Methods
  • formatPredictions()
  • getShapesWithImages()
  • getImagesFromPDF()
• Contribute to this repo
• License

Installation

Installing with npm

npm install --save react-mindee-js

installing with yarn

yarn add react-mindee-js

Use cases

Hello world

Display an image with one custom shape

import {
  AnnotationViewer,
  createPolygonFromCoordinates
} from "react-mindee-js";

const HelloWorld = () => {
  const [image, setImage] = useState(null)

  const shapes = [
    {
      polygon: createPolygonFromCoordinates(
        [
          [0.66, 0.3],
          [0.12, 0.5],
          [0.52, 0.8],
          [0.82, 0.24]
        ]
      ),
      color: "#fd3246",
      data:"This is red shape",
      index: 0,
      active: {
        color: '#00f0f0',
        lineWidth: 3
      }
    }
  ]

  return (
    <div>
      <input type="file" onChange={e => setImage(URL.createObjectURL(e.target.files[0]))}/>
      {
        image && <div style={{ height: '500px', width: '400px'}}>
          <AnnotationViewer
            shapes={shapes}
            image={image}
            onShapeClick={shape => console.log('shapeClicked', shape)}
            onShapeHover={shape => console.log('shapeHovered', shape)}
          />
        </div>
      }
    </div>
  );
}
export default HelloWorld;

Handle multi-pages pdfs navigation

Handle multi-pages pdfs navigation using the AnnotationSidebar component.

import {
  AnnotationViewer,
  createPolygonFromCoordinates,
  getImagesFromPDF,
  AnnotationSidebar
} from "react-mindee-js";

const MultiPagesPdf = () => {
  const [pdfPages, setPdfPages] = useState([])
  const [currentPdfPageIndex, setCurrentPdfPageIndex] = useState(0)

  const pagesShapes = [
    [
      {
        polygon: createPolygonFromCoordinates(
          [
            [0.66, 0.3],
            [0.12, 0.5],
            [0.52, 0.8],
            [0.82, 0.24]
          ]
        ),
        color: "#fd3246",
        data: "This is first page shape",
        index: 0,
        active: {
          color: '#00f0f0',
          lineWidth: 3
        }
      }
    ],
    [
      {
        polygon: createPolygonFromCoordinates(
          [
            [0.26, 0.3],
            [0.32, 0.5],
            [0.52, 0.64],
            [0.52, 0.28]
          ]
        ),
        color: "#fd3246",
        data: "This is second page shape",
        index: 0,
        active: {
          color: '#00f0f0',
          lineWidth: 3
        }
      }
    ]
  ]

  const handleUpload = (e) => {
    getImagesFromPDF(URL.createObjectURL(e.target.files[0]))
      .then((_pdfPages) => {setPdfPages(_pdfPages)}
    )
  }

  return(
    <div>
      <input type="file" onChange={e => handleUpload(e)}/>
      {
        pdfPages.length > 0 &&
        <div style={{ height: '500px', width: '400px'}}>
          <AnnotationViewer
            image={pdfPages[currentPdfPageIndex]}
            shapes={pagesShapes[currentPdfPageIndex]}
          />
          <AnnotationSidebar
            items={pdfPages}
            activeIndex={currentPdfPageIndex}
            onChange={setCurrentPdfPageIndex}
          />
        </div>
      }
    </div>
  )
}
export default MultiPagesPdf;

Add lens

Create easily a zoomed area wherever you need in your app.

import {
  AnnotationViewer,
  createPolygonFromCoordinates,
  Point,
  AnnotationLens
 } from "react-mindee-js";

const Lens = () => {
  const [image, setImage] = useState(null)
  const [lensProps, setLensProps] = useState({
    cursorPosition: new Point(),
    selectedShape: null,
  })

  const shapes = [
    {
      polygon: createPolygonFromCoordinates(
        [
          [0.66, 0.3],
          [0.12, 0.5],
          [0.52, 0.8],
          [0.82, 0.24]
        ]
      ),
      color: "#fd3246",
      data:"This is red shape",
      index: 0,
      active: {
        color: '#00f0f0',
        lineWidth: 3
      }
    }
  ]

  return (
    <div>
      <input type="file" onChange={e => setImage(URL.createObjectURL(e.target.files[0]))}/>
      {
        image && <div style={{position: 'relative'}}>
          <div style={{ height: '500px', width: '400px'}}>
            <AnnotationViewer
              shapes={shapes}
              image={image}
              onShapeClick={shape => console.log('shapeClicked', shape)}
              onShapeHover={shape => console.log('shapeHovered', shape)}
              getLensProps={setLensProps}
            />
          </div>
          <div
            style={{
              position: "absolute",
              left: "10px",
              top: "10px",
              height: '100px',
              width: '100px'
            }}
          >
            <AnnotationLens
              image={image}
              shapes={shapes}
              {...lensProps}
            />
          </div>
        </div>
      }
    </div>
  );
}
export default Lens;

Mindee APIs helpers

To help you prototype using mindee APIs, you can directly use the formatPrediction method to create shapes from a raw Mindee API response.

There is also a fake response exported from the library so you don't have to code your API calls to get started.

import {
  AnnotationViewer,
  fakeResponse,
  formatPrediction
} from "react-mindee-js";

const Mindee = () => {
  const [image, setImage] = useState(null)

  const shapes = formatPrediction(fakeResponse.predictions[0])

  return (
    <div>
      <input type="file" onChange={e => setImage(URL.createObjectURL(e.target.files[0]))}/>
      {
        image && <div style={{ height: '500px', width: '400px'}}>
          <AnnotationViewer
            shapes={shapes}
            image={image}
            onShapeClick={shape => console.log('shapeClicked', shape)}
            onShapeHover={shape => console.log('shapeHovered', shape)}
          />
        </div>
      }
    </div>
  );
}
export default Mindee;

Components

AnnotationViewer

import { AnnotationViewer } from 'react-mindee-js';

<AnnotationViewer
    shapes={shapes}
    image={image}
    onShapeClick={shape => console.log('shapeClicked', shape)}
    onShapeHover={shape => console.log('shapeHovered', shape)}
/>

The annotation viewer is the main component on which you can feed an image or pdf, and display a list of shapes (items).

Props

• shapes: List of objects to display on the canvas. Each object must have the following attributes:
  • polygon: A list of relative vertices passed to createPolygonFromCoordinates
  • color: Hex color for borders and background shape
  • index: Number used for identifying the shape. When many shapes have the same index, hovering one of them will activate the others.
  • active: An object to set the style of the shape when hovered

    active: {
      color: '#00ff00',
      lineWidth: 3
    }
    

  You can add any custom attributes to pass data through your shapes.

  import { createPolygonFromCoordinates } from "react-mindee-js";
  
  const shapes = [
    {
      polygon: createPolygonFromCoordinates(
        [
          [0.66, 0.3],
          [0.12, 0.5],
          [0.52, 0.8],
          [0.82, 0.24]
        ]
      ),
      color: "#fd3246",
      data:"This is red shape",
      index: 0,
      active: {
        color: '#00ff00',
        lineWidth: 3
      }
    }
  ]
  

• image: Image URL or base64
• className: Container className props. The default canvas style will fill his parent height and width
• onShapeHover: Binded to onHover of a shape
• onShapeClick: Binded to onClick of a shape
• getLensProps: Returns AnnotationLens props to be passed if you use one.

AnnotationLens

The AnnotationLens component is a zoomed window displaying the area arround the user's pointer.

It works combined with a AnnotationViewer, check out the related use case below for an example.

<AnnotationLens
  image={image}
  shapes={shapes}
  cursorPosition={cursorPosition}
  selectedShape={selectedShape}
/>

Props

• image: The source image passed to the AnnotationViewer
• cursorPosition: Cursor position on the AnnotationViewer. You can get it using the getLensProps from the AnnotationViewer.
• selectedShape: Selected shape on the AnnotationViewer . You can get it using the getLensProps from the AnnotationViewer.
• shapes (optional): Pass the same shapes passed to the AnnotationViewer if you want them to be displayed on the Lens.

AnnotationSidebar

The AnnotationSidebar is a vertical carousel displaying thumbnails of pdf pages.

<AnnotationSidebar
    items={pdfPages}
    activeIndex={currentPdfPageIndex}
    onChange={setCurrentPdfPageIndex}
/>

Props

• items: List of images
• activeIndex: Active page number
• onChange: Triggered when clicking on a thumbnail

Fullscreen

The Fullscreen component enables a fullscreen mode for its children.

<Fullscreen
  customButton={(onClick) => <div onClick={onClick}>My custom Button</div>}
>
  <AnnotationViewer
    items={items}
    source={file.preview}
    type={file.type}
  />
</Fullscreen>

props

• customButton: A custom component binding the onClick method to enable the full screen mode

Methods

formatPredictions

This function formats raw predictions from Mindee APIs to feed the AnnotationViewer shapes easily.

const shapes = formatPrediction(mindeeAPIResponse.predictions[0])

async getShapesWithImages

This function generate sub images from a source image corresponding to shapes.

getShapesWithImages(file.preview, styled_shapes).then(
    _shapesWithImages => {
      // Do something with shapesWithImages
    }
)

async getImagesFromPDF

This function generate images from a pdf.

getImagesFromPDF(pdfFile)
  .then((_pdfPages) => {
    // Do something with images pdf pages
  }
)

• pdfFile: object URL of a pdf file

Contribute to this repo

Feel free to use github to submit issues, pull requests or general feedback. You can also visit our website or drop us an email.

Please read our Contributing section before contributing.

License

GPLv3 © mindee

Changelog

v1.1.2 (05/08/2020)

Fix

• 🐛 Rework promises in getImagesFromPDF function