github.com/ethicalml/awesome-production-machine-learning
This repository contains a curated list of awesome open source libraries that will help you deploy, monitor, version, scale and secure your production machine learning 🚀
| This 10 minute video provides an overview of the motivations for machine learning operations as well as a high level overview on some of the tools in this repo. This newer video covers the an updated 2022 version of the state of MLOps |
|
| You can join the Machine Learning Engineer newsletter. Join over 10,000 ML professionals and enthusiasts who receive weekly curated articles & tutorials on production Machine Learning. |
|
| Also check out the Awesome Artificial Intelligence Guidelines List, where we aim to map the landscape of "Frameworks", "Codes of Ethics", "Guidelines", "Regulations", etc related to Artificial Intelligence. |
|
- An open-source bias audit toolkit for data scientists, machine learning researchers, and policymakers to audit machine learning models for discrimination and bias, and to make informed and equitable decisions around developing and deploying predictive risk-assessment tools.
- Interpretability and explainability of data and machine learning models including a comprehensive set of algorithms that cover different dimensions of explanations along with proxy explainability metrics.
- A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.
- Alibi is an open source Python library aimed at machine learning model inspection and interpretation. The initial focus on the library is on black-box, instance based model explanations.
- Code for the paper "High precision model agnostic explanations", a model-agnostic system that explains the behaviour of complex models with high-precision rules called anchors.
- model interpretability and understanding library for PyTorch developed by Facebook. It contains general purpose implementations of integrated gradients, saliency maps, smoothgrad, vargrad and others for PyTorch models.
- Example of using classifier-agnostic saliency map extraction on ImageNet presented on the paper "Classifier-agnostic saliency map extraction".
- An adversarial example library for constructing attacks, building defenses, and benchmarking both. A python library to benchmark system's vulnerability to adversarial examples.
- Python script for model agnostic contrastive/counterfactual explanations for machine learning. Accompanying code for the paper "Contrastive Explanations with Local Foil Trees".
- Codebase that contains the methods in the paper "Learning important features through propagating activation differences". Here is the slides and the video of the 15 minute talk given at ICML.
- This is the code required to run the Deep Visualization Toolbox, as well as to generate the neuron-by-neuron visualizations using regularized optimisation. The toolbox and methods are described casually here and more formally in this paper.
- "Explain Like I'm 5" is a Python package which helps to debug machine learning classifiers and explain their predictions.
- Facets contains two robust visualizations to aid in understanding and analyzing machine learning datasets. Get a sense of the shape of each feature of your dataset using Facets Overview, or explore individual observations using Facets Dive.
- Fairlearn is a python toolkit to assess and mitigate unfairness in machine learning models.
- FairML is a python toolbox auditing the machine learning models for bias.
- This repository is meant to facilitate the benchmarking of fairness aware machine learning algorithms based on this paper.
- The tool supports teams in evaluating, improving, and comparing models for fairness concerns in partnership with the broader Tensorflow toolkit.
- An attention-based summarized post-hoc explanations for detection and identification of bias in data. We propose a global explanation and introduce a step-by-step framework on how to detect and test bias. Python package for image data.
- An open-source library for analyzing Keras models visually by methods such as DeepTaylor-Decomposition, PatternNet, Saliency Maps, and Integrated Gradients.
- This repository provides code for implementing integrated gradients for networks with image inputs.
- InterpretML is an open-source package for training interpretable models and explaining blackbox systems.
- keras-vis is a high-level toolkit for visualizing and debugging your trained keras neural net models. Currently supported visualizations include: Activation maximization, Saliency maps, Class activation maps.
- Code for replicating the experiments in the paper "Learning to Explain: An Information-Theoretic Perspective on Model Interpretation" at ICML 2018.
- A python framework for self-supervised learning on images. The learned representations can be used to analyze the distribution in unlabeled data and rebalance datasets.
- A Pytorch based framework that breaks down machine learning problems into smaller blocks that can be glued together seamlessly with an objective to build predictive models with one line of code.
- Local Interpretable Model-agnostic Explanations for machine learning models.
- LOFO (Leave One Feature Out) Importance calculates the importances of a set of features based on a metric of choice, for a model of choice, by iteratively removing each feature from the set, and evaluating the performance of the model, with a validation scheme of choice, based on the chosen metric.
- MindsDB is an Explainable AutoML framework for developers. With MindsDB you can build, train and use state of the art ML models in as simple as one line of code.
- A Python package for AutoML on tabular data with feature engineering, hyper-parameters tuning, explanations and automatic documentation.
- Viewer for neural network, deep learning and machine learning models.
- A model agnostic tool for decomposition of predictions from black boxes. Break Down Table shows contributions of every variable to a final prediction.
- Toolkit for auditing and mitigating bias and fairness of machine learning systems
- SHapley Additive exPlanations is a unified approach to explain the output of any machine learning model.
- Shapash is a Python library that provides several types of visualization that display explicit labels that everyone can understand.
- Skater is a unified framework to enable Model Interpretation for all forms of model to help one build an Interpretable machine learning system often needed for real world use-cases.
- TensorFlow Model Analysis (TFMA) is a library for evaluating TensorFlow models. It allows users to evaluate their models on large amounts of data in a distributed manner, using the same metrics defined in their trainer.
- themis-ml is a Python library built on top of pandas and sklearn that implements fairness-aware machine learning algorithms.
- Themis is a testing-based approach for measuring discrimination in a software system.
- Package for interpreting scikit-learn's decision tree and random forest predictions. Allows decomposing each prediction into bias and feature contribution components as described here.
- An easy-to-use interface for expanding understanding of a black-box classification or regression ML model.
- Tools for WoE Transformation mostly used in ScoreCard Model for credit rating
- An eXplainability toolbox for machine learning.
- BastionLab is a framework for confidential data science collaboration. It uses Confidential Computing, Access control data science, and Differential Privacy to enable data scientists to remotely perform data exploration, statistics, and training on confidential data while ensuring maximal privacy for data owners.
- Concrete-ML is a Privacy-Preserving Machine Learning (PPML) open-source set of tools built on top of The Concrete Framework by Zama. It aims to simplify the use of fully homomorphic encryption (FHE) for data scientists to help them automatically turn machine learning models into their homomorphic equivalent.
- Fedlearner is collaborative machine learning framework that enables joint modeling of data distributed between institutions.
- FATE (Federated AI Technology Enabler) is the world's first industrial grade federated learning open source framework to enable enterprises and institutions to collaborate on data while protecting data security and privacy.
- FedML provides a research and production integrated edge-cloud platform for Federated/Distributed Machine Learning at anywhere at any scale.
- Flower is a Federated Learning Framework with a unified approach. It enables the federation of any ML workload, with any ML framework, and any programming language.
- This is a C++ library of ε-differentially private algorithms, which can be used to produce aggregate statistics over numeric data sets containing private or sensitive information.
- The Intel HE transformer for nGraph is a Homomorphic Encryption (HE) backend to the Intel nGraph Compiler, Intel's graph compiler for Artificial Neural Networks.
- Microsoft SEAL is an easy-to-use open-source (MIT licensed) homomorphic encryption library developed by the Cryptography Research group at Microsoft.
- OpenFL is a Python framework for Federated Learning. OpenFL is designed to be a flexible, extensible and easily learnable tool for data scientists. OpenFL is developed by Intel Internet of Things Group (IOTG) and Intel Labs.
- A Python library for secure, private Deep Learning. PySyft decouples private data from model training, using Multi-Party Computation (MPC) within PyTorch.
- A privacy-preserving framework based on TensorFlow with customized backend Operations using Multi-Party Computation (MPC). Rosetta reuses the APIs of TensorFlow and allows to transfer original TensorFlow codes into a privacy-preserving manner with minimal changes.
- Substra is an open-source framework for privacy-preserving, traceable and collaborative Machine Learning.
- A Python library that includes implementations of TensorFlow optimizers for training machine learning models with differential privacy.
- A Framework for Confidential Machine Learning on Encrypted Data in TensorFlow.
- A super-easy way to record, search and compare AI experiments.
- High-level utils for PyTorch DL & RL research. It was developed with a focus on reproducibility, fast experimentation and code/ideas reusing.
- Auto-Magical Experiment Manager & Version Control for AI (previously Trains).
- CodaLab Worksheets is a collaborative platform for reproducible research that allows researchers to run, manage, and share their experiments in the cloud. It helps researchers ensure that their runs are reproducible and consistent.
- A git fork that allows for version management of models.
- An open-source platform and cross-platform desktop application to execute, track, and debug modern machine learning experiments.
- Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a git repository.
- Easy to use logger and automatic version controller made for data scientists who write ML code.
- Open source toolkit that automates and optimizes machine learning experiments.
- Version control for tensor data, git-like semantics on numerical data with high speed and efficiency.
- Version control for machine learning.
- Repeatable, atomic and versioned data lake on top of object storage.
- Open source platform to manage the ML lifecycle, including experimentation, reproducibility and deployment.
- An open-source system to version machine learning models including their ingredients code, data, config, and environment and to track ML metadata across the model lifecycle.
- An open-source Python library that allows you to version, export, and save a machine learning model to your cloud storage provider.
- Docker for Your ML/DL Models Based on OCI Artifacts.
- A platform for reproducible and scalable machine learning and deep learning on kubernetes - (Video).
- Versioning, reproducibility and deployment of data and models.
- Tool to help you configure, organize, log and reproduce machine learning experiments.
- Model management framework which minimizes the overhead involved with scheduling, running, monitoring and managing artifacts of your machine learning experiments.
- A graph database management system that stores data like git.
- Accelerate abstracts exactly and only the boilerplate code related to multi-GPU/TPU/mixed-precision and leaves the rest of your code unchanged.
- Aqueduct enables you to easily define, run, and manage AI & ML tasks on any cloud infrastructure.
- Continuous Machine Learning (CML) is an open-source library for implementing continuous integration & delivery (CI/CD) in machine learning projects.
- Deep learning training platform with integrated support for distributed training, hyperparameter tuning, and model management (supports Tensorflow and Pytorch).
- Machine learning development environment for data science and AI/ML engineering teams.
- Hopsworks is a data-intensive platform for the design and operation of machine learning pipelines that includes a Feature Store - (Video).
- A cloud native platform for machine learning based on Google’s internal machine learning pipelines.
- Standardisation of pipeline and model serialization for Spark, Tensorflow and sklearn.
- 
nos is an open-source platform to efficiently run AI workloads on Kubernetes, increasing GPU utilization and reducing infrastructure and operational costs.
- TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.
- Production scale vision AI platform, with fully integrated components for model building, automated labeling, data processing and model training pipelines.
- Platform that provides complete AI model training and resource management capabilities.
) - low-code library for training and deploying models (scikit-learn, XGBoost, LightGBM, spaCy)
- Platform to build resource-intensive pipelines with simple Python.
- Skaffold is a command line tool that facilitates continuous development for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters.
- Run LLMs, AI, and batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution -- all with a simple interface.
- A Data Streaming Library for Efficient Neural Network Training.
- Production oriented configuration framework for ML based on TensorFlow, incl. monitoring and model version management.
- TonY is a framework to natively run deep learning jobs on Apache Hadoop. It currently supports TensorFlow, PyTorch, MXNet and Horovod.
- ZenML is an extensible, open-source MLOps framework to create reproducible ML pipelines with a focus on automated metadata tracking, caching, and many integrations to other tools.
- Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.
- BentoML is an open source framework for high performance ML model serving.
- Cortex is an open source platform for deploying machine learning models—trained with any framework—as production web services. No DevOps required.
- Deepchecks is an open source package for comprehensively validating your machine learning models and data with minimal effort during development, deployment or in production.
- Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain.
- Evidently helps analyze machine learning models during development, validation, or production monitoring. The tool generates interactive reports from pandas DataFrame.
- Cloud-native machine learning model server.
- Quality Assurance for AI models. Open-source platform to help organizations increase the efficiency of their AI development workflow, eliminate risks of AI biases and ensure robust, reliable & ethical AI models.
- Helicone is an observability platform for LLMs.
- Open source model management cluster for deploying, serving and monitoring machine learning models and ad-hoc algorithms with a FaaS architecture.
- An Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere.
- Cloud native search framework that supports to use deep learning/state of the art AI models for search.
- Serverless framework to deploy and monitor machine learning models in Kubernetes - (Video).
- Langfuse is an open source observability & analytics solution for LLM-based applications.
- LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.
- Observability & analytics for AI apps and agents.
- LocalAI is a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing.
- A lightweight library which allows to transpile trained classic machine learning models into a native code of C, Java, Go, R, PHP, Dart, Haskell, Rust and many other programming languages.
- Version and deploy your ML models following GitOps principles.
- An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more.
- a lightweight, open-source Python tool to get "bolt-on" observability in ML pipelines.
- MLWatcher is a python agent that records a large variety of time-serie metrics of your running ML classification algorithm. It enables you to monitor in real time.
- A model server for Apache MXNet from Amazon Web Services that is able to run MXNet models as well as Gluon models (Amazon's SageMaker runs a custom version of MMS under the hood).
- An open source library to estimate post-deployment model performance (without access to targets). Capable of fully capturing the impact of data drift on performance.
- A rust-powered and multi-stage pipelined model server which offers dynamic batching and more. Super easy to implement and deploy as micro-services.
- A high-performance "serverless" framework focused on data, I/O, and compute intensive workloads. It is well integrated with popular data science tools, such as Jupyter and Kubeflow; supports a variety of data and streaming sources; and supports execution over CPUs and GPUs.
- ONNX Runtime is a cross-platform inference and training machine-learning accelerator.
- REST web service for the true real-time scoring (< 1 ms) of Scikit-Learn, R and Apache Spark models.
- OpenVINO is an open-source toolkit for optimizing and deploying AI inference.
- Phoenix is an open source ML observability in a notebook to validate, monitor and fine tune your generative LLM, CV and tabular models.
- PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.
- An open source Machine Learning Server built on top of a state-of-the-art open source stack for developers and data scientists to create predictive engines for any machine learning task.
- A Redis module for serving tensors and executing deep learning models. Expect changes in the API and internals.
- Open source platform for deploying and monitoring machine learning models in kubernetes - (Video).
- skops is a Python library helping you share your scikit-learn based models and put them in production.
- Serving Thousands of Concurrent LoRA Adapters.
- Open source SDK that provides a unified interface to multiple MLOps projects that enable data scientists to deploy and productionise machine learning systems.
- High-performant framework to serve Tensorflow models via grpc protocol able to handle 100k requests per second per core.
- TorchServe is a flexible and easy to use tool for serving PyTorch models.
- Transformer-deploy is an efficient, scalable and enterprise-grade CPU/GPU inference server for Hugging Face transformer models.
- Triton is a high performance open source serving software to deploy AI models from any framework on GPU & CPU while maximizing utilization.
- TruLens provides a set of tools for developing and monitoring neural nets, including large language models.
- UnionML is an open source MLOps framework that aims to reduce the boilerplate and friction that comes with building models and deploying them to production.
- vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs
- A toolbox to generate adversarial examples that fool neural networks in PaddlePaddle, PyTorch, Caffe2, MxNet, Keras, TensorFlow, and Advbox can benchmark the robustness of machine learning models.
- think TensorFlow Playground, but for Adversarial Examples! A visualization tool designed for learning and teaching - the attack library is limited in size, but it has a nice front-end to it with buttons you can press!
- library for adversarial attacks / defenses specifically for PyTorch.
AirBnB's library to generate text that reads the same to a human but passes adversarial classifiers. - library for testing adversarial attacks / defenses maintained by some of the most important names in adversarial ML, namely Ian Goodfellow (ex-Google Brain, now Apple) and Nicolas Papernot (Google Brain). Comes with some nice tutorials!
- Counterfit is a command-line tool and generic automation layer for assessing the security of machine learning systems.
- another systematic tool for attacking and defending deep learning models.
- second biggest adversarial library. Has an even longer list of attacks - but no defenses or evaluation metrics. Geared more towards computer vision. Code easier to understand / modify than ART - also better for exploring blackbox attacks on surrogate models.
- ART provides tools that enable developers and researchers to defend and evaluate Machine Learning models and applications against the adversarial threats of Evasion, Poisoning, Extraction, and Inference.
- A library for running membership inference attacks (MIA) against machine learning models.
- NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.
- OpenAttack is a Python-based textual adversarial attack toolkit, which handles the whole process of textual adversarial attacking, including preprocessing text, accessing the victim model, generating adversarial examples and evaluation.
- another robustness resource maintained by some of the leading names in adversarial ML. They specifically focus on defenses, and onesa standardized adversarial robustness benchmark.
- plausible looking adversarial examples for text generation.
- Library and experiments for attacking machine learning in discrete domains using graph search.
- Automated feature, model, and hyperparameter selection for tabular, image, and text data on top of popular machine learning libraries (Scikit-Learn, LightGBM, CatBoost, PyTorch, MXNet).
- AutoML library for Keras based on "Auto-Keras: Efficient Neural Architecture Search with Network Morphism".
- Automatic feature and model search with code generation in Python, on top of common data science libraries (tensorflow, sklearn, etc.).
- Framework to automate algorithm and hyperparameter tuning for sklearn.
- Efficient Neural Architecture Search (ENAS) in PyTorch based on this paper.
- Efficient Neural Architecture search via parameter sharing(ENAS) micro search Tensorflow code for windows user.
- Feature-engine is a Python library that contains several transformers to engineer features for use in machine learning models.
- An open source framework for automated feature engineering.
- FLAML is a fast library for automated machine learning & tuning.
- A feature pre-processing framework in Go that matches functionality of sklearn.
- A Kubernetes-based system for Hyperparameter Tuning and Neural Architecture Search.
- Keras Tuner is an easy-to-use, distributable hyperparameter optimisation framework that solves the pain points of performing a hyperparameter search. Keras Tuner makes it easy to define a search space and leverage included algorithms to find the best hyperparameter values.
- Asynchronous, directed Hyperparameter search and parallel ablation studies on Apache Spark - (Video).
- Basic implementation of Controller RNN from Neural Architecture Search with Reinforcement Learning and Learning Transferable Architectures for Scalable Image Recognition.
- NNI (Neural Network Intelligence) is a toolkit to help users run automated machine learning (AutoML) experiments.
- Optuna is an automatic hyperparameter optimisation software framework, particularly designed for machine learning.
- OSS Vizier is a Python-based service for black-box optimisation and research, one of the first hyperparameter tuning services designed to work at scale.
Use evolutionary algorithms instead of gridsearch in scikit-learn.
- Automation of sklearn pipeline creation (including feature selection, pre-processor, etc.). - Automatic extraction of relevant features from time series.
- Free automated data & feature enrichment library for machine learning: automatically searches through thousands of ready-to-use features from public and community shared data sources and enriches your training dataset with only the accuracy improving features.
- Data Pipeline framework built in Python, including scheduler, DAG definition and a UI for visualisation.
- Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, and system mediation logic.
- Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).
- Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.
- Visual programming editor for building Spark and PySpark pipelines.
- BatchFlow helps data scientists conveniently work with random or sequential batches of your data and define data processing and machine learning workflows for large datasets.
- ETL framework for Python 3.5+ with focus on simple atomic operations working concurrently on rows of data.
- More of a job scheduler for Mesos than ETL pipeline.
- Unified interface for constructing and managing machine learning workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.
- A python library that allows for building complex data science workflows on Python.
- DALL·E Flow is an interactive workflow for generating high-definition images from text prompt.
- A data orchestrator for machine learning, analytics, and ETL.
- DBND is an agile pipeline framework that helps data engineering teams track and orchestrate their data processes.
- ETL tool for running transformations inside data warehouses.
- Lyft’s Cloud Native Machine Learning and Data Processing Platform - (Demo).
- Job orchestration engine to interface and trigger the execution of jobs from Hadoop-based systems.
- Wrapper of the data pipeline Luigi.
- Hamilton is a micro-orchestration framework for defining dataflows. Runs anywhere python runs (e.g. jupyter, fastAPI, spark, ray, dask). Brings software engineering best practices without you knowing it. Use it to define feature engineering transforms, end-to-end model pipelines, and LLM workflows. It complements macro-orchestration systems (e.g. kedro, luigi, airflow, dbt, etc.) as it replaces the code within those macro tasks.
- Instill VDP (Versatile Data Pipeline) aims to streamline the data processing pipelines from inception to completion.
- Kedro is a workflow development tool that helps you build data pipelines that are robust, scalable, deployable, reproducible and versioned. Visualization of the kedro workflows can be done by kedro-viz.
- Ludwig is a declarative machine learning framework that makes it easy to define machine learning pipelines using a simple and flexible data-driven configuration system.
- Luigi is a Python module that helps you build complex pipelines of batch jobs, handling dependency resolution, workflow management, visualisation, etc..
- A framework for data scientists to easily build and manage real-life data science projects.
- A framework for building neat pipelines, providing the right abstractions to chain your data transformation and prediction steps with data streaming, as well as doing hyperparameter searches (AutoML).
- Workflow scheduler for Hadoop jobs.
- Open source distributed processing framework build on Kubernetes focused mainly on dynamic building of production machine learning pipelines - (Video).
- Based on Kedro and MLflow. Full comparison is found here.
- The fastest way to build data pipelines. Develop iteratively, deploy anywhere.
- Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.
- A simple Spark-powered ETL framework that helps you structure your ETL projects, modularize your data transformation logic and speed up your development.
- Workflow management system for reproducible and scalable data analyses.
- General-purpose machine learning pipeline for generating embedding vectors using one or many ML models.
- Argilla helps domain experts and data teams to build better NLP datasets in less time.
- Baal is an active learning library that supports both industrial applications and research usecases.
- Web-based text annotation tool for Named-Entity-Recogntion task.
- Python library for data-centric AI. Can automatically: find mislabeled data, detect outliers, estimate consensus + annotator-quality for multi-annotator datasets, suggest which data is best to (re)label next.
- Web-based image segmentation tool for object detection, localization and keypoints
- OpenCV's web-based annotation tool for both VIDEOS and images for computer algorithms.
- Open source text annotation tools for humans, providing functionality for sentiment analysis, named entity recognition, and machine translation.
- Image labelling tool with support for collaboration, supporting bounding box, polygon, line, point labelling, label export, etc.
- Image annotation tool for bounding boxes with auto-suggestion and extensibility for plugins.
- Multi-domain data labeling and annotation tool with standardized output format.
- Open source graphical image annotation tool writen in Python using QT for graphical interface focusing primarily on bounding boxes.
- Free to use online tool for labelling photos. Prepared labels can be downloaded in one of multiple supported formats.
- A collaborative framework for annotating medical datasets using crowdsourcing.
- modAL is an active learning framework designed with modularity, flexibility and extensibility in mind.
- Open source tool for labelling images with support for labels, edges, as well as image resizing and zooming in.
- Image annotation tool with ability to "colour" on the images to select labels for segmentation. Process is semi-automated with the watershed marked algorithm of OpenCV
- The data scientist's open-source choice to scale, assess and maintain natural language data.
- Open-source tool for tracking, exploring, and labeling data for AI projects.
- Synthetic Data Vault (SDV) is a Synthetic Data Generation ecosystem of libraries that allows users to easily learn single-table, multi-table and timeseries datasets to later on generate new Synthetic Data that has the same format and statistical properties as the original dataset.
- Hitachi's Open source tool for labelling camera and LIDAR data.
- Snorkel is a system for quickly generating training data with weak supervision.
- superintendent provides an ipywidget-based interactive labelling tool for your data.
- YData Synthetic is a package to generate synthetic tabular and time-series data leveraging the state of the art generative models.
- Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.
- ArangoML Pipeline is a common and extensible Metadata Layer for Machine Learning Pipelines which allows Data Scientists and DataOps to manage all information related to their ML pipeline in one place.
- Apache Atlas framework is an extensible set of core foundational governance services – enabling enterprises to effectively and efficiently meet their compliance requirements within Hadoop and allows integration with the whole enterprise data ecosystem.
- DataHub is LinkedIn's generalized metadata search & discovery tool.
- Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata.
- Metacat is a unified metadata exploration API service. Metacat focusses on solving these problems: 1) federated views of metadata systems; 2) arbitrary metadata storage about data sets; 3) metadata discovery.
- a library for recording and retrieving metadata associated with ML developer and data scientist workflows.
- streamlines and automates generation of Model Cards.
- AIStore is a lightweight object storage system with the capability to linearly scale out with each added storage node and a special focus on petascale deep learning.
- A virtual distributed storage system that bridges the gab between computation frameworks and storage systems.
- In-memory columnar representation of data compatible with Pandas, Hadoop-based systems, etc..
- A high performance real-time analytics database. Check this article for introduction.
- A memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale - Demo.
- On-disk columnar representation of data compatible with Pandas, Hadoop-based systems, etc..
- A realtime distributed OLAP datastore. Comparison of the open source OLAP systems for big data: ClickHouse, Druid, and Pinot is found here.
- A Bayesian database table for querying the probable implications of data as easily as SQL databases query the data itself. - (Video)
- BayesDB is an AI-native embedding database.
- ClickHouse is an open source column oriented database management system.
- Delta Lake is a storage layer that brings scalable, ACID transactions to Apache Spark and other big-data engines.
- NoSQL interface for Postgres that allows for object interaction to data stored.
- GPTCache is a library for creating semantic cache for large language model queries.
- HDFS-compatible file system with scale-out strongly consistent metadata.
Scalable datastore for metrics, events, and real-time analytics.
Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.
Marqo is an end-to-end vector search engine.
pgvector helps with vector similarity search for Postgres.
PostgresML is a machine learning extension for PostgreSQL that enables you to perform training and inference on text and tabular data using SQL queries.
Simple, safe way to store and distribute tensors.
An open-source time-series SQL database optimized for fast ingest and complex queries packaged as a PostgreSQL extension - (Video).
- A low-latency vector search engine (GraphQL, RESTful) with out-of-the-box support for different media types. Modules include Semantic Search, Q&A, Classification, Customizable Models (PyTorch/TensorFlow/Keras), and more.
- Python implementation of chunked, compressed, N-dimensional arrays designed for use in parallel computing.
- A unified Data Analytics and AI platform for distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray.
- Bagua is a performant and flexible distributed training framework for PyTorch, providing a faster alternative to PyTorch DDP and Horovod. It supports advanced distributed training algorithms such as quantization and decentralization.
Apache Beam is a unified programming model for Batch and Streaming.
- Deep learning framework on top of Spark/Hadoop to distribute data and computations across a HDFS system.
- A unified deep learning system for big model era, which helps users to efficiently and quickly deploy large AI model training and inference.
- Distributed parallel processing framework for Pandas and NumPy computations - (Video).
- A novel evolutionary computation framework for rapid prototyping and testing of ideas. It seeks to make algorithms explicit and data structures transparent. It works in perfect harmony with parallelisation mechanisms such as multiprocessing and SCOOP.
- A deep learning optimization library (lightweight PyTorch wrapper) that makes distributed training easy, efficient, and effective.
- Flexible and powerful tensor operations for readable and reliable code.
- Distributed computing library for modern computer clusters from Uber.
- A fast, flexible machine learning library written entirely in C++ from the Facebook AI Research and the creators of Torch, TensorFlow, Eigen and Deep Speech.
- Decentralized deep learning in PyTorch.
- Uber's distributed training framework for TensorFlow, Keras, and PyTorch.
- LightGBM is a gradient boosting framework that uses tree based learning algorithms.
- Scientific computing framework build on top of pywren to enable numpy-like distributed computations.
- Answer the question of the "cloud button" for python function execution. It's a framework that abstracts AWS Lambda to enable data scientists to execute any Python function - (Video).
- Lightweight PyTorch research framework that allows you to easily scale your models to GPUs and TPUs and use all the latest best practices, without the engineering boilerplate - (Video).
- Ray is a flexible, high-performance distributed execution framework for machine learning (VIDEO).
- TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
Vespa is an engine for low-latency computation over large data sets.
- Cross-framework solution to convert, visualize and diagnose deep neural network models.
- Open Neural Network Exchange Format.
- TensorStore is an open-source C++ and Python software library designed for storage and manipulation of large multi-dimensional arrays.
- BindsNET is a spiking neural network simulation library geared towards the development of biologically inspired algorithms for machine learning.
- BrainCog (Brain-inspired Cognitive Intelligence Engine) is a brain-inspired spiking neural network based platform for Brain-inspired Artificial Intelligence and simulating brains at multiple scales.
- Composer is a PyTorch library that enables you to train neural networks faster, at lower cost, and to higher accuracy.
- Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
- cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions that share compatible APIs with other RAPIDS projects.
- An implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it.
- A neural network library and ecosystem for JAX designed for flexibility.
- A tensor library for machine learning that you can efficiently run GPT-2 and GPT-J inference on the CPU.
- Fast scalable Machine Learning platform for smarter applications: Deep Learning, Gradient Boosting & XGBoost, Random Forest, Generalized Linear Modeling (Logistic Regression, Elastic Net), K-Means, PCA, Stacked Ensembles, Automatic Machine Learning (AutoML), etc..
- Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more.
- MLX is an array framework for machine learning on Apple silicon.
- Speed up your Pandas workflows by changing a single line of code.
- Nebullvm is an ecosystem of plug and play modules to boost the performances of your AI systems. The optimisation modules are stack-agnostic and work with any library. They are designed to be easily integrated into your system, providing a quick and seamless boost to its performance.
- Nevergrad is a gradient-free optimisation platform.
- Norse aims to exploit the advantages of bio-inspired neural components, which are sparse and event-driven - a fundamental difference from artificial neural networks.
- A compiler for Python array and numerical functions.
Optimised tools for group-indexing operations: aggregated sum and more
- OpenFlamingo is an open-source framework for training large multimodal models. - OpenVINO is an open-source toolkit for optimizing and deploying AI inference.
- Optimum is an extension of Transformers and Diffusers, providing a set of optimization tools enabling maximum efficiency to train and run models on targeted hardware, while keeping things easy to use.
- Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters.
- snnTorch is a deep and online learning library with spiking neural networks.
- Sonnet is a library built on top of TensorFlow 2 designed to provide simple, composable abstractions for machine learning research.
- Tensor2Tensor is a library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
The torchkeras library is a simple tool for training neural network in pytorch jusk in a keras style.
- TorchOpt is an efficient library for differentiable optimization built upon PyTorch.
Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.
- Blazing fast, lightweight and mobile phone-enabled Vulkan compute framework optimized for advanced GPU data processing usecases.
- High-performance runtime for data analytics applications, Here is an interview with Weld’s main contributor.
- XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable.
- Open source stream processing framework with powerful stream and batch processing capabilities.
- Distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.
- Distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.
- Flexible Python-centric stateful stream processing framework built on top of Rust engine.
- A modern broker-agnostic streaming Python framework supporting Apache Kafka, RabbitMQ and NATS protocols, inspired by FastAPI and easily integratable with other web frameworks.
- Streaming library built on top of Python's Asyncio library using the async kafka client inspired by the kafka streaming library.
- Micro-batch processing for streams using the apache spark framework as a backend supporting stateful exactly-once semantics.
- Kafka client library for buliding applications and microservices where the input and output are stored in kafka clusters.
- A Python toolkit for rule-based/unsupervised anomaly detection in time series.
- alibi-detect is a Python package focused on outlier, adversarial and concept drift detection.
- Outlier detection in heterogeneous datasets using automatic tuple expansion. Check this paper for further details.
- A library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
- Outlier Exposure (OE) is a method for improving anomaly detection performance in deep learning models. Paper
- A Python Toolbox for Scalable Outlier Detection (Anomaly Detection).
- An Acceleration System for Large-scale Anomaly/Outlier Detection.
- Library for exploring and validating machine learning data.
- TextAttack is a Python framework for adversarial attacks, data augmentation, and model training in NLP.
- TODS is a full-stack automated machine learning system for outlier detection on multivariate time-series data.
- A tool for building feature stores which allows you to transform your raw data into beautiful features.
- Feast (Feature Store) is an open source feature store for machine learning. Feast is the fastest path to manage existing infrastructure to productionize analytic data for model training and online inference.
- A virtual featurestore. Plug-&-play with your existing infra. Data Scientist approved. Discovery, Governance, Lineage, & Collaboration just a pip install away. Supports pandas, Python, spark, SQL + integrations with major cloud vendors. - Offline/Online Feature Store for ML (Video).
- ivory defines a specification for how to store feature data and provides a set of tools for querying it. It does not provide any tooling for producing feature data in the first place. All ivory commands run as MapReduce jobs so it assumed that feature data is maintained on HDFS.
- Veri is a Feature Label Store. Feature Label store allows storing features as keys and labels as values. Querying values is only possible with knn using features. Veri also supports creating sub sample spaces of data by default.
- Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
- Web interface python sandbox environments for reproducible development
- All-in-one web IDE for machine learning and data science. Combines Jupyter, VS Code, Tensorflow, and many other tools/libraries into one Docker image.
- .NET Interactive takes the power of .NET and embeds it into your interactive experiences.
- Papermill is a library for parameterizing notebooks and executing them like Python scripts.
- Polynote is an experimental polyglot notebook environment. Currently, it supports Scala and Python (with or without Spark), SQL, and Vega.
- The rmarkdown package is a next generation implementation of R Markdown based on Pandoc. - Stencila is a platform for creating, collaborating on, and sharing data driven content. Content that is transparent and reproducible.
- Voilà turns Jupyter notebooks into standalone web applications that can e.g. be used as dashboards.
- Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given query point.
- CLIP-as-service is a low-latency high-scalability service for embedding images and text. It can be easily integrated as a microservice into neural search solutions.
- DocArray is a library for nested, unstructured, multimodal data in transit, including text, image, audio, video, 3D mesh, etc. It allows deep-learning engineers to efficiently process, embed, search, recommend, store, and transfer multimodal data with a Pythonic API.
- Faiss is a library for efficient similarity search and clustering of dense vectors.
- Finetuner provides an effective way to improve performance on neural search tasks.
- NGT provides commands and a library for performing high-speed approximate nearest neighbor searches against a large volume of data in high dimensional vector data space.
- Non-Metric Space Library (NMSLIB): An efficient similarity search library and a toolkit for evaluation of k-NN methods for generic non-metric spaces.
- An open source vector similarity search engine with extended filtering support.
- AutoAWQ is an easy-to-use package for 4-bit quantized models.
- An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
- Activation-aware Weight Quantization for LLM Compression and Acceleration.
- Accurate Post-training Quantization of Generative Pretrained Transformers.
- Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks.
- Deep Lake is a data infrastructure optimized for computer vision.
- Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation algorithms.
- iGibson is a simulation environment providing fast visual rendering and physics simulation based on Bullet.
- KerasCV is a library of modular computer vision oriented Keras components.
- LAVIS is a deep learning library for LAnguage-and-VISion intelligence research and applications.
- MMDetection is an open source object detection toolbox based on PyTorch.
- SuperGradients is an open-source library for training PyTorch-based computer vision models.
- VISSL is FAIR's library of extensible, modular and scalable components for SOTA Self-Supervised Learning with images.
- We write your reusable computer vision tools. Whether you need to load your dataset from your hard drive, draw detections on an image or video, or count how many detections are in a zone. You can count on us!
- Built atop Zalando Research's Flair and Hugging Face's Transformers library, AdaptNLP provides Machine Learning Researchers and Scientists a modular and adaptive approach to a variety of NLP tasks with an Easy API for training, inference, and deploying NLP-based microservices.
- Blackstone is a spaCy model and library for processing long-form, unstructured legal text. Blackstone is an experimental research project from the Incorporated Council of Law Reporting for England and Wales' research lab, ICLR&D.
- Coqui STT is a fast, open-source, multi-platform, deep-learning toolkit for training and deploying speech-to-text models.
- A Conditional Transformer Language Model for Controllable Generation released by SalesForce.
- A framework for programming with foundation models.
- Dust assists in the design and deployment of large language model apps.
- ESPnet is an end-to-end speech processing toolkit.
- PyTorch original implementation of Cross-lingual Language Model Pretraining which includes BERT, XLM, NMT, XNLI, PKM, etc..
- FastChat is an open platform for training, serving, and evaluating large language model based chatbots.
- Simple framework for state-of-the-art NLP developed by Zalando which builds directly on PyTorch.
- FlexGen is a high-throughput generation engine for running large language models with limited GPU memory.
- GluonNLP is a toolkit that enables easy text preprocessing, datasets loading and neural models building to help you speed up your Natural Language Processing (NLP) research.
- Gretel Synthetics is a synthetic data generators for structured and unstructured text, featuring differentially private learning.
- Grover is a model for Neural Fake News -- both generation and detection. However, it probably can also be used for other generation tasks.
- Guardrails is a package that lets a user add structure, type and quality guarantees to the outputs of large language models.
- h2oGPT is an open source generative AI, gives organizations like yours the power to own large language models while preserving your data ownership.
- Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-3 and alike). Haystack offers production-ready tools to quickly build ChatGPT-like question answering, semantic search, text generation, and more.
- ICE is a Python library and trace visualizer for language model programs.
- Kashgari is a simple and powerful NLP Transfer learning framework, build a state-of-art model in 5 minutes for named entity recognition (NER), part-of-speech tagging (PoS), and text classification tasks.
- Lamini is an LLM engine for rapidly customizing models.
- LangChain assists in building applications with LLMs through composability.
- LlamaIndex (GPT Index) is a data framework for your LLM application.
- LLaMA is intended as a minimal, hackable and readable example to load LLaMA (arXiv) models and run inference.
- LMFlow is an extensible, convenient, and efficient toolbox for finetuning large machine learning models.
- Megatron-LM is a highly optimized and efficient library for training large language models.
- MLC LLM is a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications, plus a productive framework for everyone to further optimize model performance for their own use cases.
- Get up and running with large language models, locally.
- A Pytorch library that allows for training and using sense2vec models, which are models that leverage the same approach than word2vec, but also leverage part-of-speech attributes for each token, which allows it to be "meaning-aware".
- Sentence Transformers provides an easy method to compute dense vector representations for sentences, paragraphs, and images.
- Industrial-strength natural language processing library built with python and cython by the explosion.ai team.
- Stability AI language models.
- A framework for building neural networks in Tensorflow, particularly sequence models.
- TensorFlow Text provides a collection of text related classes and ops ready to use with TensorFlow 2.0.
- Huggingface's library of state-of-the-art pretrained models for Natural Language Processing (NLP).
- Large Language Model Text Generation Inference under TFOIL license.
- trlX is a distributed training framework designed from the ground up to focus on fine-tuning large language models with reinforcement learning using either a provided reward function or a reward-labeled dataset.
- YouTokenToMe is an unsupervised text tokenizer focused on computational efficiency. It currently implements fast Byte Pair Encoding (BPE).
- Acme is a library of reinforcement learning (RL) building blocks that strives to expose simple, efficient, and readable agents.
- AI-Optimizer is a next-generation deep reinforcement learning suit, providing rich algorithm libraries ranging from model-free to model-based RL algorithms, from single-agent to multi-agent algorithms. Moreover, AI-Optimizer contains a flexible and easy-to-use distributed training framework for efficient policy training.
- ALF is a reinforcement learning framework emphasizing on the flexibility and easiness of implementing complex algorithms involving many different components.
- AlpacaFarm is a simulation framework for methods that learn from human feedback.
- CityLearn is an open source OpenAI Gym environment for the implementation of Multi-Agent Reinforcement Learning (RL) for building energy coordination and demand response in cities.
- CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch.
- CompilerGym is a library of easy to use and performant reinforcement learning environments for compiler tasks.
- d3rlpy is an offline deep reinforcement learning library for practitioners and researchers.
- DIAMBRA Arena is a software package featuring a collection of high-quality environments for Reinforcement Learning research and experimentation.
- Dopamine is a research framework for fast prototyping of reinforcement learning algorithms. It aims to fill the need for a small, easily grokked codebase in which users can freely experiment with wild ideas (speculative research).
- EvoTorch is an open source evolutionary computation library developed at NNAISENSE, built on top of PyTorch.
- FinRL is the first open-source framework to demonstrate the great potential of financial reinforcement learning.
- garage is a toolkit for developing and evaluating reinforcement learning algorithms, and an accompanying library of state-of-the-art implementations built using that toolkit.
- Gymnasium is an open source Python library for developing and comparing reinforcement learning algorithms by providing a standard API to communicate between learning algorithms and environments, as well as a standard set of environments compliant with that API.
- Gymnasium-Robotics contains a collection of Reinforcement Learning robotic environments that use the Gymansium API. The environments run with the MuJoCo physics engine and the maintained mujoco python bindings.
- Jumanji is a suite of Reinforcement Learning (RL) environments written in JAX providing clean, hardware-accelerated environments for industry-driven research.
- MALib is a parallel framework of population-based learning nested with reinforcement learning methods. MALib provides higher-level abstractions of MARL training paradigms, which enables efficient code reuse and flexible deployments on different distributed computing paradigms.
- MARLlib is a comprehensive Multi-Agent Reinforcement Learning algorithm library based on RLlib. It provides MARL research community with a unified platform for building, training, and evaluating MARL algorithms.
- Mava is a framework for distributed multi-agent reinforcement learning in JAX.
- Melting Pot is a suite of test scenarios for multi-agent reinforcement learning.
- MetaDrive is a driving simulator that composes diverse driving scenarios for generalizable RL.
- The Minigrid library contains a collection of discrete grid-world environments to conduct research on Reinforcement Learning. The environments follow the Gymnasium standard API and they are designed to be lightweight, fast, and easily customizable.
- MiniHack is a sandbox framework for easily designing rich and diverse environments for Reinforcement Learning
- MiniWorld is a minimalistic 3D interior environment simulator for reinforcement learning & robotics research.
- ML-Agents is an open-source project that enables games and simulations to serve as environments for training intelligent agents.
- MushroomRL is a Python Reinforcement Learning (RL) library whose modularity allows to easily use well-known Python libraries for tensor computation (e.g. PyTorch, Tensorflow) and RL benchmarks (e.g. OpenAI Gym, PyBullet, Deepmind Control Suite).
- PARL is a flexible and high-efficient reinforcement learning framework.
- PettingZoo is a Python library for conducting research in multi-agent reinforcement learning, akin to a multi-agent version of Gymnasium. - RLlib is an open-source library for reinforcement learning (RL), offering support for production-level, highly distributed RL workloads, while maintaining unified and simple APIs for a large variety of industry applications.
- RLMeta is a flexible lightweight research framework for Distributed Reinforcement Learning based on PyTorch and moolib
- Safety-Gymnasium is a highly scalable and customizable safe reinforcement learning environment library.
- skrl is an open-source modular library for Reinforcement Learning written in Python (using PyTorch) and designed with a focus on readability, simplicity, and transparency of algorithm implementation.
- A fork of OpenAI Baselines, implementations of reinforcement learning algorithms.
- SuperSuit introduces a collection of small functions which can wrap reinforcement learning environments to do preprocessing ('microwrappers').
- A reliable, scalable and easy to use TensorFlow library for contextual bandits and reinforcement learning.
- Train transformer language models with reinforcement learning.
- Bokeh is an interactive visualization library for Python that enables beautiful and meaningful visual presentation of data in modern web browsers.
- geoplotlib is a python toolbox for visualizing geographical data and making maps.
- An implementation of the grammar of graphics for R.
- Quickly create and share demos of models - by only writing Python. Debug models interactively in your browser, get feedback from collaborators, and generate public links without deploying anything.
- Kangas is a tool for exploring, analyzing, and visualizing large-scale multimedia data. It provides a straightforward Python API for logging large tables of data, along with an intuitive visual interface for performing complex queries against your dataset.
- A Python 2D plotting library which produces publication-quality figures in a variety of hardcopy formats and interactive environments across platforms.
- missingno provides a small toolset of flexible and easy-to-use missing data visualizations and utilities that allows you to get a quick visual summary of the completeness (or lack thereof) of your dataset.
- This repository is inspired by ICEbox. The goal is to visualize the impact of certain features towards model prediction for any supervised learning algorithm.
Streaming pivot visualization via WebAssembly.
- PixieDust is a productivity tool for Python or Scala notebooks, which lets a developer encapsulate business logic into something easy for your customers to consume.
- An interactive, open source, and browser-based graphing library for Python.
- Python Individual Conditional Expectation Plot Toolbox.
- pygal is a dynamic SVG charting library written in Python.
- Redash is anopen source visualisation framework that is built to allow easy access to big datasets leveraging multiple backends.
- Seaborn is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics.
- Spotlight helps you to identify critical data segments and model failure modes. It enables you to build and maintain reliable machine learning models by curating high-quality datasets.
- Streamlit lets you create apps for your machine learning projects with deceptively simple Python scripts. It supports hot-reloading, so your app updates live as you edit and save your file.
- A modern, enterprise-ready business intelligence web application.
- Write TensorBoard events with simple function call.
- A visualization toolkit for machine learning experimentation that makes it easy to host, track, and share ML experiments.
- yellowbrick is a matplotlib-based model evaluation plots for scikit-learn and other machine learning libraries.
- EasyRec is a framework for large scale recommendation algorithms.
- Gorse aims to be a universal open-source recommender system that can be quickly introduced into a wide variety of online services.
- Implicit provides fast Python implementations of several different popular recommendation algorithms for implicit feedback datasets
- LightFM is a Python implementation of a number of popular recommendation algorithms for both implicit and explicit feedback
- NVTabular is a feature engineering and preprocessing library for tabular data that is designed to easily manipulate terabyte scale datasets and train deep learning (DL) based recommender systems.
- NVIDIA Merlin is an open source library providing end-to-end GPU-accelerated recommender systems, from feature engineering and preprocessing to training deep learning models and running inference in production.
- Surprise is a Python scikit for building and analyzing recommender systems that deal with explicit rating data.
- A Comprehensive Benchmark to Evaluate LLMs as Agents.
- An automatic evaluator for instruction-following language models.
- Evaluation tool for LLM QA chains.
- A framework for the evaluation of autoregressive code generation language models.
- The Beyond the Imitation Game Benchmark (BIG-bench) is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities.
- D4RL is an open-source benchmark for offline reinforcement learning.
- DeepEval is a simple-to-use, open-source evaluation framework for LLM applications.
- A benchmarking and visualization tool for adversarial ML.
- EvalAI is an open source platform for evaluating and comparing AI algorithms at scale.
- Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.
- Evaluate is a library that makes evaluating and comparing models and reporting their performance easier and more standardized.
- Holistic Evaluation of Language Models (HELM) is a benchmark framework to increase the transparency of language models.
- LM-Eval is a framework for few-shot evaluation of autoregressive language models.
- Lucid is a collection of infrastructure and tools for research in neural network interpretability.
- Meta-World is an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of many distinct robotic manipulation tasks.
- Multi-Modality Arena is an evaluation platform for large multi-modality models.
- Massive Text Embedding Benchmark (MTEB) is a comprehensive benchmark of text embeddings.
- OmniSafe is a comprehensive and reliable benchmark for safe reinforcement learning, covering a multitude of SafeRL domains and delivering a new suite of testing environments.
- OpenCompass is an LLM evaluation platform, supporting a wide range of models (LLaMA, LLaMa2, ChatGLM2, ChatGPT, Claude, etc) over 50+ datasets.
- A zoo for models tuned for OpenCV DNN with benchmarks on different platforms.
- A unified multi-backend utility for benchmarking Transformers and Diffusers with support for Optimum's arsenal of hardware optimizations/quantization schemes.
- Overcooked-AI is a benchmark environment for fully cooperative human-AI task performance, based on the wildly popular video game Overcooked.
- PandaLM aims to provide reproducible and automated comparisons between different large language models.
- Large language model evaluation and workflow framework from Phase AI.
- Ragas is a framework that helps evaluate Retrieval Augmented Generation (RAG) pipelines.
- Recommenders contains benchmark and best practices for building recommendation systems, provided as Jupyter notebooks.
- RLeXplore provides stable baselines of exploration methods in reinforcement learning.
- SafePO-Baselines is a benchmark repository for safe reinforcement learning algorithms.
- UpTrain is an open-source tool to evaluate LLM applications.
- Neptune is a lightweight solution designed for: 1) experiment tracking; 2) model registry; 3) ML runs live monitoring.
- Machine learning experiment tracking, dataset versioning, hyperparameter search, visualization, and collaboration.FAQs
Unknown package
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.
