Socket
Socket
Sign inDemoInstall

nncf

Package Overview
Dependencies
31
Maintainers
3
Alerts
File Explorer

Install Socket

Detect and block malicious and high-risk dependencies

Install

nncf

Neural Networks Compression Framework

    2.8.1

Maintainers
3

Readme

Neural Network Compression Framework (NNCF)

Key FeaturesInstallationDocumentationUsageTutorials and SamplesThird-party integrationModel Zoo

GitHub Release Website Apache License Version 2.0 PyPI Downloads

Neural Network Compression Framework (NNCF) provides a suite of post-training and training-time algorithms for neural networks inference optimization in OpenVINO™ with minimal accuracy drop.

NNCF is designed to work with models from PyTorch, TensorFlow, ONNX and OpenVINO™.

NNCF provides samples that demonstrate the usage of compression algorithms for different use cases and models. See compression results achievable with the NNCF-powered samples at Model Zoo page.

The framework is organized as a Python* package that can be built and used in a standalone mode. The framework architecture is unified to make it easy to add different compression algorithms for both PyTorch and TensorFlow deep learning frameworks.

Key Features

Post-Training Compression Algorithms

Compression algorithmOpenVINOPyTorchTensorFlowONNX
Post-Training QuantizationSupportedSupportedSupportedSupported
Weights CompressionSupportedSupportedNot supportedNot supported

Training-Time Compression Algorithms

Compression algorithmPyTorchTensorFlow
Quantization Aware TrainingSupportedSupported
Mixed-Precision QuantizationSupportedNot supported
BinarizationSupportedNot supported
SparsitySupportedSupported
Filter pruningSupportedSupported
Movement pruningExperimentalNot supported
  • Automatic, configurable model graph transformation to obtain the compressed model.

    NOTE: Limited support for TensorFlow models. The models created using Sequential or Keras Functional API are only supported.

  • Common interface for compression methods.
  • GPU-accelerated layers for faster compressed model fine-tuning.
  • Distributed training support.
  • Git patch for prominent third-party repository (huggingface-transformers) demonstrating the process of integrating NNCF into custom training pipelines
  • Seamless combination of pruning, sparsity and quantization algorithms. Please refer to optimum-intel for examples of joint (movement) pruning, quantization and distillation (JPQD), end-to-end from NNCF optimization to compressed OpenVINO IR.
  • Exporting PyTorch compressed models to ONNX* checkpoints and TensorFlow compressed models to SavedModel or Frozen Graph format, ready to use with OpenVINO™ toolkit.
  • Support for Accuracy-Aware model training pipelines via the Adaptive Compression Level Training and Early Exit Training.

Documentation

This documentation covers detailed information about NNCF algorithms and functions needed for the contribution to NNCF.

The latest user documentation for NNCF is available here.

NNCF API documentation can be found here.

Usage

Post-Training Quantization

The NNCF PTQ is the simplest way to apply 8-bit quantization. To run the algorithm you only need your model and a small (~300 samples) calibration dataset.

OpenVINO is the preferred backend to run PTQ with, and PyTorch, TensorFlow and ONNX are also supported.

OpenVINO
import nncf
import openvino.runtime as ov
import torch
from torchvision import datasets, transforms

# Instantiate your uncompressed model
model = ov.Core().read_model("/model_path")

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1)

# Step 1: Initialize transformation function
def transform_fn(data_item):
    images, _ = data_item
    return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)
PyTorch
import nncf
import torch
from torchvision import datasets, models

# Instantiate your uncompressed model
model = models.mobilenet_v2()

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset)

# Step 1: Initialize the transformation function
def transform_fn(data_item):
    images, _ = data_item
    return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)

TensorFlow
import nncf
import tensorflow as tf
import tensorflow_datasets as tfds

# Instantiate your uncompressed model
model = tf.keras.applications.MobileNetV2()

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = tfds.load("/path", split="validation",
                        shuffle_files=False, as_supervised=True)

# Step 1: Initialize transformation function
def transform_fn(data_item):
    images, _ = data_item
    return images

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(val_dataset, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(model, calibration_dataset)
ONNX
import onnx
import nncf
import torch
from torchvision import datasets

# Instantiate your uncompressed model
onnx_model = onnx.load_model("/model_path")

# Provide validation part of the dataset to collect statistics needed for the compression algorithm
val_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
dataset_loader = torch.utils.data.DataLoader(val_dataset, batch_size=1)

# Step 1: Initialize transformation function
input_name = onnx_model.graph.input[0].name
def transform_fn(data_item):
    images, _ = data_item
    return {input_name: images.numpy()}

# Step 2: Initialize NNCF Dataset
calibration_dataset = nncf.Dataset(dataset_loader, transform_fn)
# Step 3: Run the quantization pipeline
quantized_model = nncf.quantize(onnx_model, calibration_dataset)

Training-Time Compression

Below is an example of Accuracy Aware Quantization pipeline where model weights and compression parameters may be fine-tuned to achieve a higher accuracy.

PyTorch
import torch
import nncf.torch  # Important - must be imported before any other external package that depends on torch

from nncf import NNCFConfig
from nncf.torch import create_compressed_model, register_default_init_args

# Instantiate your uncompressed model
from torchvision.models.resnet import resnet50
model = resnet50()

# Load a configuration file to specify compression
nncf_config = NNCFConfig.from_json("resnet50_int8.json")

# Provide data loaders for compression algorithm initialization, if necessary
import torchvision.datasets as datasets
representative_dataset = datasets.ImageFolder("/path", transform=transforms.Compose([transforms.ToTensor()]))
init_loader = torch.utils.data.DataLoader(representative_dataset)
nncf_config = register_default_init_args(nncf_config, init_loader)

# Apply the specified compression algorithms to the model
compression_ctrl, compressed_model = create_compressed_model(model, nncf_config)

# Now use compressed_model as a usual torch.nn.Module
# to fine-tune compression parameters along with the model weights

# ... the rest of the usual PyTorch-powered training pipeline

# Export to ONNX or .pth when done fine-tuning
compression_ctrl.export_model("compressed_model.onnx")
torch.save(compressed_model.state_dict(), "compressed_model.pth")

NOTE (PyTorch): Due to the way NNCF works within the PyTorch backend, import nncf must be done before any other import of torch in your package or in third-party packages that your code utilizes, otherwise the compression may be applied incompletely.

Tensorflow
import tensorflow as tf

from nncf import NNCFConfig
from nncf.tensorflow import create_compressed_model, register_default_init_args

# Instantiate your uncompressed model
from tensorflow.keras.applications import ResNet50
model = ResNet50()

# Load a configuration file to specify compression
nncf_config = NNCFConfig.from_json("resnet50_int8.json")

# Provide dataset for compression algorithm initialization
representative_dataset = tf.data.Dataset.list_files("/path/*.jpeg")
nncf_config = register_default_init_args(nncf_config, representative_dataset, batch_size=1)

# Apply the specified compression algorithms to the model
compression_ctrl, compressed_model = create_compressed_model(model, nncf_config)

# Now use compressed_model as a usual Keras model
# to fine-tune compression parameters along with the model weights

# ... the rest of the usual TensorFlow-powered training pipeline

# Export to Frozen Graph, TensorFlow SavedModel or .h5  when done fine-tuning
compression_ctrl.export_model("compressed_model.pb", save_format="frozen_graph")

For a more detailed description of NNCF usage in your training code, see this tutorial.

Demos, Tutorials and Samples

For a quicker start with NNCF-powered compression, try sample notebooks and scripts presented below.

Jupyter* Notebook Tutorials and Demos

A collection of ready-to-run Jupyter* notebooks tutorials and demos are available to explain and display NNCF compression algorithms for optimizing models for inference with the OpenVINO Toolkit.

Notebook Tutorial NameCompression AlgorithmBackendDomain
BERT Quantization
Colab
Post-Training QuantizationOpenVINONLP
MONAI Segmentation Model Quantization
Binder
Post-Training QuantizationOpenVINOSegmentation
PyTorch Model QuantizationPost-Training QuantizationPyTorchImage Classification
TensorFlow Model QuantizationPost-Training QuantizationTensorflowImage Classification
Migrating from POT to NNCF
Colab
Post-Training QuantizationOpenVINOObject detection
Quantization with Accuracy ControlPost-Training Quantization with Accuracy ControlOpenVINOSpeech-to-Text,
Object Detection
PyTorch Training-Time CompressionTraining-Time CompressionPyTorchImage Classification
TensorFlow Training-Time CompressionTraining-Time CompressionTensorflowImage Classification
Joint Pruning, Quantization and Distillation for BERTJoint Pruning, Quantization and DistillationOpenVINONLP

Below is a list of notebooks demonstrating OpenVINO conversion and inference together with NNCF compression for models from various domains.

Demo ModelCompression AlgorithmBackendDomain
YOLOv8
Colab
Post-Training QuantizationOpenVINOObject Detection,
KeyPoint Detection,
Instance Segmentation
YOLOv7Post-Training QuantizationOpenVINOObject Detection
EfficientSAMPost-Training QuantizationOpenVINOImage Segmentation
Segment Anything ModelPost-Training QuantizationOpenVINOImage Segmentation
OneFormerPost-Training QuantizationOpenVINOImage Segmentation
InstructPix2PixPost-Training QuantizationOpenVINOImage-to-Image
CLIPPost-Training QuantizationOpenVINOImage-to-Text
BLIPPost-Training QuantizationOpenVINOImage-to-Text
Segmind-VegaRTPost-Training QuantizationOpenVINOText-to-Image
Latent Consistency ModelPost-Training QuantizationOpenVINOText-to-Image
WürstchenPost-Training QuantizationOpenVINOText-to-Image
ControlNet QR Code MonsterPost-Training QuantizationOpenVINOText-to-Image
SDXL-turboPost-Training QuantizationOpenVINOText-to-Image,
Image-to-Image
DeepFloyd IFPost-Training Quantization,
Weight Compression
OpenVINOText-to-Image,
Image-to-Image
ImageBindPost-Training QuantizationOpenVINOMulti-Modal Retrieval
Distil-WhisperPost-Training QuantizationOpenVINOSpeech-to-Text
Whisper
Colab
Post-Training QuantizationOpenVINOSpeech-to-Text
MMS Speech RecognitionPost-Training QuantizationOpenVINOSpeech-to-Text
Grammar Error CorrectionPost-Training QuantizationOpenVINONLP, Grammar Correction
LLM Instruction FollowingWeight CompressionOpenVINONLP, Instruction Following
Dolly 2.0Weight CompressionOpenVINONLP, Instruction Following
Stable-Zephyr-3bWeight CompressionOpenVINONLP, Chat Bot
LLM Chat BotsWeight CompressionOpenVINONLP, Chat Bot

Post-Training Quantization Examples

Compact scripts demonstrating quantization and corresponding inference speed boost:

Example NameCompression AlgorithmBackendDomain
OpenVINO MobileNetV2Post-Training QuantizationOpenVINOImage Classification
OpenVINO YOLOv8Post-Training QuantizationOpenVINOObject Detection
OpenVINO YOLOv8 QwAСPost-Training Quantization with Accuracy ControlOpenVINOObject Detection
OpenVINO Anomaly ClassificationPost-Training Quantization with Accuracy ControlOpenVINOAnomaly Classification
PyTorch MobileNetV2Post-Training QuantizationPyTorchImage Classification
PyTorch SSDPost-Training QuantizationPyTorchObject Detection
TensorFlow MobileNetV2Post-Training QuantizationTensorFlowImage Classification
ONNX MobileNetV2Post-Training QuantizationONNXImage Classification

Training-Time Compression Examples

These examples provide full pipelines including compression, training and inference for classification, detection and segmentation tasks.

Example NameCompression AlgorithmBackendDomain
PyTorch Image ClassificationTraining-Time CompressionPyTorchImage Classification
PyTorch Object DetectionTraining-Time CompressionPyTorchObject Detection
PyTorch Semantic SegmentationTraining-Time CompressionPyTorchSemantic Segmentation
TensorFlow Image ClassificationTraining-Time CompressionTensorFlowImage Classification
TensorFlow Object DetectionTraining-Time CompressionTensorFlowObject Detection
TensorFlow Instance SegmentationTraining-Time CompressionTensorFlowInstance Segmentation

Third-party repository integration

NNCF may be straightforwardly integrated into training/evaluation pipelines of third-party repositories.

Used by

  • OpenVINO Training Extensions

    NNCF is integrated into OpenVINO Training Extensions as model optimization backend. So you can train, optimize and export new models based on the available model templates as well as run exported models with OpenVINO.

  • HuggingFace Optimum Intel

    NNCF is used as a compression backend within the renowned transformers repository in HuggingFace Optimum Intel.

Installation Guide

For detailed installation instructions please refer to the Installation page.

NNCF can be installed as a regular PyPI package via pip:

pip install nncf

If you want to install both NNCF and the supported PyTorch version in one line, you can do this by simply running:

pip install nncf[torch]

Other viable options besides [torch] are [tf], [onnx] and [openvino].

NNCF is also available via conda:

conda install -c conda-forge nncf

System requirements

  • Ubuntu* 18.04 or later (64-bit)
  • Python* 3.7 or later
  • Supported frameworks:
    • PyTorch* >=2.0, <2.2
    • TensorFlow* >=2.8.4, <=2.12.1
    • ONNX* ~=1.13.1
    • OpenVINO* >=2022.3.0

This repository is tested on Python* 3.8.10, PyTorch* 2.1.2 (NVidia CUDA* Toolkit 11.8) and TensorFlow* 2.12.1 (NVidia CUDA* Toolkit 11.8).

NNCF Compressed Model Zoo

List of models and compression results for them can be found at our Model Zoo page.

Citing

@article{kozlov2020neural,
    title =   {Neural network compression framework for fast model inference},
    author =  {Kozlov, Alexander and Lazarevich, Ivan and Shamporov, Vasily and Lyalyushkin, Nikolay and Gorbachev, Yury},
    journal = {arXiv preprint arXiv:2002.08679},
    year =    {2020}
}

Contributing Guide

Refer to the CONTRIBUTING.md file for guidelines on contributions to the NNCF repository.

Telemetry

NNCF as part of the OpenVINO™ toolkit collects anonymous usage data for the purpose of improving OpenVINO™ tools. You can opt-out at any time by running the following command in the Python environment where you have NNCF installed:

opt_in_out --opt_out

More information is available at https://docs.openvino.ai/latest/openvino_docs_telemetry_information.html.

Keywords

FAQs


Did you know?

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc