Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip.
Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali. Infinity is developed under MIT License.
pip install infinity_clientv2 cli, including --api-keypip install infinity-emb[all]
After your pip install, with your venv active, you can run the CLI directly.
infinity_emb v2 --model-id BAAI/bge-small-en-v1.5
Check the v2 --help command to get a description for all parameters.
infinity_emb v2 --help
Instead of installing the CLI via pip, you may also use docker to run michaelf34/infinity.
Make sure you mount your accelerator ( i.e. install nvidia-docker and activate with --gpus all).
port=7997
model1=michaelfeil/bge-small-en-v1.5
model2=mixedbread-ai/mxbai-rerank-xsmall-v1
volume=$PWD/data
docker run -it --gpus all \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest \
v2 \
--model-id $model1 \
--model-id $model2 \
--port $port
The cache path inside the docker container is set by the environment variable HF_HOME.
docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-cpu \
v2 \
--engine optimum \
--model-id $model1 \
--model-id $model2 \
--port $port
Visit Docs for more info.
This image has support for:
pip install flash-attn package when using Pytorch.docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-trt-onnx \
v2 \
--engine optimum \
--device cuda \
--model-id $model1 \
--port $port
Since infinity_emb>=0.0.34, you can use cli v2 method to launch multiple models at the same time.
Checkout infinity_emb v2 --help for all args and validation.
Multiple Model CLI Playbook:
v2 --model-id model/id1 --model-id/id2 --batch-size 8 --batch-size 4. This will create two models model/id1 and model/id2;: INFINITY_MODEL_ID="model/id1;model/id2;" && INFINITY_BATCH_SIZE="8;4;"--model-id length, v2 --model-id model/id1 --model-id/id2 --batch-size 8 making both models have batch-size 8.--model-id + API requests are routed to the --served-model-name/--model-idEnvironment variables start with INFINITY_{UPPER_CASE_SNAKE_CASE} and often match the --{lower-case-kebab-case} cli arguments.
The following two are equivalent:
infinity_emb v2 --model-id BAAI/bge-base-en-v1.5export INFINITY_MODEL_ID="BAAI/bge-base-en-v1.5" && infinity_emb v2Multiple arguments can be used via ; syntax: INFINITY_MODEL_ID="model/id1;model/id2;"
With the command --engine torch the model must be compatible with https://github.com/UKPLab/sentence-transformers/ and AutoModel
With the command --engine optimum, there must be an onnx file. Models from https://huggingface.co/Xenova are recommended.
With the command --engine ctranslate2
- only BERT models are supported.
See which telemetry is collected: https://michaelfeil.eu/infinity/main/telemetry/
# Disable
export INFINITY_ANONYMOUS_USAGE_STATS="0"
Infinity aims to be the inference server supporting most functionality for embeddings, reranking and related RAG tasks. The following Infinity tests 15+ architectures and all of the below cases in the Github CI. Click on the sections below to find tasks and validated example models.
Text embeddings measure the relatedness of text strings. Embeddings are used for search, clustering, recommendations. Think about a private deployed version of openai's text embeddings. https://platform.openai.com/docs/guides/embeddings
Tested embedding models:
Other models:
Tested reranking models:
Other reranking models:
Image<->text models can be used for e.g. photo-gallery search, where users can type in keywords to find photos, or use a photo to find related images. Audio<->text models are less popular, and can be e.g. used to find music songs based on a text description or related music songs.
Tested image<->text models:
config.jsonTested audio<->text models:
Not supported:
For usage via the RestAPI, late-interaction embeddings may best be transported via base64 encoding.
Example notebook: https://colab.research.google.com/drive/14FqLc0N_z92_VgL_zygWV5pJZkaskyk7?usp=sharing
Tested colbert models:
For usage via the RestAPI, late-interaction embeddings may best be transported via base64 encoding.
Example notebook: https://colab.research.google.com/drive/14FqLc0N_z92_VgL_zygWV5pJZkaskyk7?usp=sharing
Tested ColPali/ColQwen models:
Tested models:
config.jsonInstead of the cli & RestAPI use infinity's interface via the Python API.
This gives you most flexibility. The Python API builds on asyncio with its await/async features, to allow concurrent processing of requests. Arguments of the CLI are also available via Python.
import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
sentences = ["Embed this is sentence via Infinity.", "Paris is in France."]
array = AsyncEngineArray.from_args([
EngineArgs(model_name_or_path = "BAAI/bge-small-en-v1.5", engine="torch", embedding_dtype="float32", dtype="auto")
])
async def embed_text(engine: AsyncEmbeddingEngine):
async with engine:
embeddings, usage = await engine.embed(sentences=sentences)
# or handle the async start / stop yourself.
await engine.astart()
embeddings, usage = await engine.embed(sentences=sentences)
await engine.astop()
asyncio.run(embed_text(array[0]))
Reranking gives you a score for similarity between a query and multiple documents. Use it in conjunction with a VectorDB+Embeddings, or as standalone for small amount of documents. Please select a model from huggingface that is a AutoModelForSequenceClassification compatible model with one class classification.
import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
query = "What is the python package infinity_emb?"
docs = ["This is a document not related to the python package infinity_emb, hence...",
"Paris is in France!",
"infinity_emb is a package for sentence embeddings and rerankings using transformer models in Python!"]
array = AsyncEmbeddingEngine.from_args(
[EngineArgs(model_name_or_path = "mixedbread-ai/mxbai-rerank-xsmall-v1", engine="torch")]
)
async def rerank(engine: AsyncEmbeddingEngine):
async with engine:
ranking, usage = await engine.rerank(query=query, docs=docs)
print(list(zip(ranking, docs)))
# or handle the async start / stop yourself.
await engine.astart()
ranking, usage = await engine.rerank(query=query, docs=docs)
await engine.astop()
asyncio.run(rerank(array[0]))
When using the CLI, use this command to launch rerankers:
infinity_emb v2 --model-id mixedbread-ai/mxbai-rerank-xsmall-v1
CLIP models are able to encode images and text at the same time.
import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
sentences = ["This is awesome.", "I am bored."]
images = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
engine_args = EngineArgs(
model_name_or_path = "wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M",
engine="torch"
)
array = AsyncEngineArray.from_args([engine_args])
async def embed(engine: AsyncEmbeddingEngine):
await engine.astart()
embeddings, usage = await engine.embed(sentences=sentences)
embeddings_image, _ = await engine.image_embed(images=images)
await engine.astop()
asyncio.run(embed(array["wkcn/TinyCLIP-ViT-8M-16-Text-3M-YFCC15M"]))
CLAP models are able to encode audio and text at the same time.
import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
import requests
import soundfile as sf
import io
sentences = ["This is awesome.", "I am bored."]
url = "https://bigsoundbank.com/UPLOAD/wav/2380.wav"
raw_bytes = requests.get(url, stream=True).content
audios = [raw_bytes]
engine_args = EngineArgs(
model_name_or_path = "laion/clap-htsat-unfused",
dtype="float32",
engine="torch"
)
array = AsyncEngineArray.from_args([engine_args])
async def embed(engine: AsyncEmbeddingEngine):
await engine.astart()
embeddings, usage = await engine.embed(sentences=sentences)
embedding_audios = await engine.audio_embed(audios=audios)
await engine.astop()
asyncio.run(embed(array["laion/clap-htsat-unfused"]))
Use text classification with Infinity's classify feature, which allows for sentiment analysis, emotion detection, and more classification tasks.
import asyncio
from infinity_emb import AsyncEngineArray, EngineArgs, AsyncEmbeddingEngine
sentences = ["This is awesome.", "I am bored."]
engine_args = EngineArgs(
model_name_or_path = "SamLowe/roberta-base-go_emotions",
engine="torch", model_warmup=True)
array = AsyncEngineArray.from_args([engine_args])
async def classifier(engine: AsyncEmbeddingEngine):
async with engine:
predictions, usage = await engine.classify(sentences=sentences)
# or handle the async start / stop yourself.
await engine.astart()
predictions, usage = await engine.classify(sentences=sentences)
await engine.astop()
asyncio.run(classifier(array["SamLowe/roberta-base-go_emotions"]))
Infinity has a generated client code for RestAPI client side usage.
If you want to call a remote infinity instance via RestAPI, install the following package locally:
pip install infinity_client
For more information, check out the Client Readme https://github.com/michaelfeil/infinity/tree/main/libs/client_infinity/infinity_client
View the docs at https:///michaelfeil.github.io/infinity on how to get started.
After startup, the Swagger Ui will be available under {url}:{port}/docs, in this case http://localhost:7997/docs. You can also find a interactive preview here: https://infinity.modal.michaelfeil.eu/docs (and https://michaelfeil-infinity.hf.space/docs)
Install via Poetry 1.8.1, Python3.11 on Ubuntu 22.04
cd libs/infinity_emb
poetry install --extras all --with lint,test
To pass the CI:
cd libs/infinity_emb
make precommit
All contributions must be made in a way to be compatible with the MIT License of this repo.
@software{feil_2023_11630143,
author = {Feil, Michael},
title = {Infinity - To Embeddings and Beyond},
month = oct,
year = 2023,
publisher = {Zenodo},
doi = {10.5281/zenodo.11630143},
url = {https://doi.org/10.5281/zenodo.11630143}
}
FAQs
Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip.
We found that infinity-emb demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Socket researchers uncover a malicious npm package posing as a tool for detecting vulnerabilities in Etherium smart contracts.
Security News
Research
A supply chain attack on Rspack's npm packages injected cryptomining malware, potentially impacting thousands of developers.
Research
Security News
Socket researchers discovered a malware campaign on npm delivering the Skuld infostealer via typosquatted packages, exposing sensitive data.