What is onnxruntime-node?
The onnxruntime-node package is a Node.js binding for the ONNX Runtime, which is a cross-platform, high-performance scoring engine for Open Neural Network Exchange (ONNX) models. It allows developers to run machine learning models in Node.js applications efficiently.
What are onnxruntime-node's main functionalities?
Load and Run ONNX Models
This feature allows you to load an ONNX model and run inference on it using the onnxruntime-node package. The code sample demonstrates how to create an inference session, prepare input data, and execute the model to get the output.
const ort = require('onnxruntime-node');
async function runModel() {
const session = await ort.InferenceSession.create('./model.onnx');
const input = { input: new ort.Tensor('float32', [1, 2, 3, 4], [2, 2]) };
const output = await session.run(input);
console.log(output);
}
runModel();
Model Optimization
This feature provides options to optimize the ONNX model for better performance. The code sample shows how to set optimization options when creating an inference session, which can lead to faster inference times.
const ort = require('onnxruntime-node');
async function optimizeModel() {
const sessionOptions = {
graphOptimizationLevel: 'all'
};
const session = await ort.InferenceSession.create('./model.onnx', sessionOptions);
console.log('Model optimized and loaded');
}
optimizeModel();
Other packages similar to onnxruntime-node
tensorflow
TensorFlow is an open-source platform for machine learning. It provides a comprehensive, flexible ecosystem of tools, libraries, and community resources that lets researchers push the state-of-the-art in ML, and developers easily build and deploy ML-powered applications. Unlike onnxruntime-node, TensorFlow is not limited to ONNX models and offers a broader range of functionalities including model training.
ml5
ml5.js is a friendly high-level interface to TensorFlow.js, designed to make machine learning accessible to a broad audience. It provides pre-trained models and simplifies the process of integrating machine learning into web applications. Compared to onnxruntime-node, ml5.js is more focused on ease of use and accessibility, particularly for beginners and educators.
ONNX Runtime Node.js Binding
ONNX Runtime Node.js binding enables Node.js applications to run ONNX model inference.
Usage
Install the latest stable version:
npm install onnxruntime-node
Refer to ONNX Runtime JavaScript examples for samples and tutorials.
Requirements
ONNXRuntime works on Node.js v16.x+ (recommend v18.x+) or Electron v15.x+ (recommend v28.x+).
The following table lists the supported versions of ONNX Runtime Node.js binding provided with pre-built binaries.
EPs/Platforms | Windows x64 | Windows arm64 | Linux x64 | Linux arm64 | MacOS x64 | MacOS arm64 |
---|
CPU | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
DirectML | ✔️ | ✔️ | ❌ | ❌ | ❌ | ❌ |
CUDA | ❌ | ❌ | ✔️[1] | ❌ | ❌ | ❌ |
To use on platforms without pre-built binaries, you can build Node.js binding from source and consume it by npm install <onnxruntime_repo_root>/js/node/
. See also instructions for building ONNX Runtime Node.js binding locally.
GPU Support
Right now, the Windows version supports only the DML provider. Linux x64 can use CUDA and TensorRT.
CUDA EP Installation
To use CUDA EP, you need to install the CUDA EP binaries. By default, the CUDA EP binaries are installed automatically when you install the package. If you want to skip the installation, you can pass the --onnxruntime-node-install-cuda=skip
flag to the installation command.
npm install onnxruntime-node --onnxruntime-node-install-cuda=skip
You can also use this flag to specify the version of the CUDA: (v11 or v12)
npm install onnxruntime-node --onnxruntime-node-install-cuda=v12
License
License information can be found here.