@annotorious/plugin-segment-anything

Annotorious SegmentAnything Plugin

A fully browser-based smart polygon selection tool for Annotorious, built on the sam2-hiera-tiny SegmentAnything model. The implementation is inspired by this blog post and demo code.

[!CAUTION] This plugin depends on the WebGPU API. It will fail to initialize if WebGPU is not avaialable or if hardware acceleration is disabled in the user's browser. If you integrate this plugin into your application, check for WebGPU support in the user's environment and provide clear feedback to the user when it is not supported.

Important: this plugin currently only supports @annotorious/openseadragon. Support for plain images (JPEG, PNG,...) is not yet available. Join the discussion if you're interested in these integrations.

Installation

npm install @annotorious/plugin-segment-anything

How to Use

Create annotations with the Annotorious SegmentAnything plugin as follows:

• Hover the mouse over the image to preview segmentation masks generated by the SAM model in real time.
• Click on a preview to select a segment and create a polygon annotation.
• Optionally, refine the polygon by marking areas to add to (click) or remove from the segment (SHIFT + click).

Note: After panning or zooming the image, the plugin needs some time to re-encode the new viewport for segmentation.

API

This plugin is designed to be headless: it provides the core segmentation functionality but requires implementers to build their own UI for interaction.

Lifecycle & Core Methods

• init() Starts initializing the plugin, including download of the SAM models (approx. 150MB). This can take some time. After download, models are stored in the browser for re-use. I.e. download only happens only once unless the user clears their browser data.

• start() Activates the plugin and enables hover previews. If not already initialized, init() will be called internally.

• stop() Deactivates the plugin and disables mask previews.

• reset() Removes the current annotation and restarts the plugin.

• restart() Stops and then starts the plugin without deleting the current annotation.

• setQueryMode(mode: 'add' | 'remove') Sets the query mode programmatically. In remove mode, clicks remove areas from the current annotation without requiring SHIFT.

• destroy() Destroys the plugin, cleaning up all resources and event listeners.

Events

The plugin emits events so the host app can respond appropriately and provide UI feedback.

Event	Parameters	Description
downloadStart	—	Model download has started.
downloadProgress	(progress: { loaded: number; total?: number; complete?: boolean })	Progress updates during model download.
initialized	—	Plugin and model initialization complete.
initError	(error: any)	Initialization failed with an error.
encodingStart	—	Started encoding the current viewport.
encodingFinished	—	Viewport encoding finished, mask preview available.
animationStart	—	User started panning or zooming the image.
animationFinished	—	User finished panning or zooming the image.
createAnnotation	(annotation: ImageAnnotation, prompt: SAM2DecoderPrompt)	A new annotation was created based on SAM output.
updateAnnotation	(annotation: ImageAnnotation, previous: ImageAnnotation, prompt: SAM2DecoderPrompt)	An existing annotation was updated/refined.
deleteAnnotation	(annotation: ImageAnnotation)	An annotation was deleted (e.g. on reset).

Usage Example

import OpenSeadragon from 'openseadragon';
import { createOSDAnnotator } from '@annotorious/openseadragon';
import { mountOpenSeadragonPlugin } from '@annotorious/plugin-segment-anything/openseadragon';

import '@annotorious/openseadragon/annotorious-openseadragon.css';

const viewer = OpenSeadragon({
  /** Your viewer config **/
});

const anno = createOSDAnnotator(viewer, { /* options */ });

// Initialize the plugin
const plugin = mountOpenSeadragonPlugin(anno);

// Start plugin initialization, including SAM model download (may take a while!)
plugin.init();

plugin.on('downloadStart', () => {
  console.log('Downloading models - this may take a while...');
});

plugin.on('downloadProgress', progress => {
  if (progress.complete)
    console.log('Download complete');
});

plugin.on('initialized', () => {
  console.log('Plugin ready');
});

plugin.on('encodingStart', () => {
  console.log('Encoding viewport...');
});

plugin.on('encodingFinished', () => {
  console.log('Ready - click to create annotation!');
});