cross-encoder/ms-marco-TinyBERT-L2-v2

Cross-Encoder for MS Marco

This model was trained on the MS Marco Passage Ranking task.

The model can be used for Information Retrieval: Given a query, encode the query will all possible passages (e.g. retrieved with ElasticSearch). Then sort the passages in a decreasing order. See SBERT.net Retrieve & Re-rank for more details. The training code is available here: SBERT.net Training MS Marco

Usage with Transformers

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model = AutoModelForSequenceClassification.from_pretrained('cross-encoder/ms-marco-TinyBERT-L2-v2')
tokenizer = AutoTokenizer.from_pretrained('cross-encoder/ms-marco-TinyBERT-L2-v2')

features = tokenizer(['How many people live in Berlin?', 'How many people live in Berlin?'], ['Berlin has a population of 3,520,031 registered inhabitants in an area of 891.82 square kilometers.', 'New York City is famous for the Metropolitan Museum of Art.'],  padding=True, truncation=True, return_tensors="pt")

model.eval()
with torch.no_grad():
    scores = model(**features).logits
    print(scores)

Usage with SentenceTransformers

The usage becomes easier when you have SentenceTransformers installed. Then, you can use the pre-trained models like this:

from sentence_transformers import CrossEncoder

model = CrossEncoder('cross-encoder/ms-marco-TinyBERT-L2-v2', max_length=512)
scores = model.predict([('Query', 'Paragraph1'), ('Query', 'Paragraph2') , ('Query', 'Paragraph3')])

Performance

In the following table, we provide various pre-trained Cross-Encoders together with their performance on the TREC Deep Learning 2019 and the MS Marco Passage Reranking dataset.

Model-Name	NDCG@10 (TREC DL 19)	MRR@10 (MS Marco Dev)	Docs / Sec
Version 2 models
cross-encoder/ms-marco-TinyBERT-L2-v2	69.84	32.56	9000
cross-encoder/ms-marco-MiniLM-L2-v2	71.01	34.85	4100
cross-encoder/ms-marco-MiniLM-L4-v2	73.04	37.70	2500
cross-encoder/ms-marco-MiniLM-L6-v2	74.30	39.01	1800
cross-encoder/ms-marco-MiniLM-L12-v2	74.31	39.02	960
Version 1 models
cross-encoder/ms-marco-TinyBERT-L2	67.43	30.15	9000
cross-encoder/ms-marco-TinyBERT-L4	68.09	34.50	2900
cross-encoder/ms-marco-TinyBERT-L6	69.57	36.13	680
cross-encoder/ms-marco-electra-base	71.99	36.41	340
Other models
nboost/pt-tinybert-msmarco	63.63	28.80	2900
nboost/pt-bert-base-uncased-msmarco	70.94	34.75	340
nboost/pt-bert-large-msmarco	73.36	36.48	100
Capreolus/electra-base-msmarco	71.23	36.89	340
amberoad/bert-multilingual-passage-reranking-msmarco	68.40	35.54	330
sebastian-hofstaetter/distilbert-cat-margin_mse-T2-msmarco	72.82	37.88	720

Note: Runtime was computed on a V100 GPU.

Changelog

Fix training code link

Fix training code link

2025-08-29 · by tomaarsen · 81d1926

Revert inadvertent config, tokenizer updates

2025-04-15 · by Tom Aarsen · d327be6

This reverts commit 15192f12d2403ed1fd7c16a8841379096c5d4a9b.

Add exported openvino model 'openvino_model_qint8_quantized.xml'

2025-04-11 · by tomaarsen · c4982bc

Add new CrossEncoder model

2025-04-11 · by tomaarsen · aab7c05

Add exported onnx model 'model_qint8_avx512_vnni.onnx'

2025-04-11 · by tomaarsen · 06bb80f

Add exported onnx model 'model_qint8_avx512.onnx'

2025-04-11 · by tomaarsen · 87f54cd

Add exported onnx model 'model_quint8_avx2.onnx'

2025-04-11 · by tomaarsen · 4ab9bb9

Add exported onnx model 'model_qint8_arm64.onnx'

2025-04-11 · by tomaarsen · 1c612c6

Add exported onnx model 'model_O4.onnx'

2025-04-11 · by tomaarsen · 4305d3b

Add exported onnx model 'model_O3.onnx'

2025-04-11 · by tomaarsen · aeeec00

Add exported onnx model 'model_O2.onnx'

2025-04-11 · by tomaarsen · c77c985

Add exported onnx model 'model_O1.onnx'

2025-04-11 · by tomaarsen · e7ec13c

Add new CrossEncoder model

2025-04-11 · by tomaarsen · 15192f1

Push tokenizer again (#5)

2025-04-08 · by tomaarsen · b25631d

- Push tokenizer again (df716dc9f28666fa90d4eb21bf73bb0e38d4ec82)

Update model metadata

2025-03-26 · by tomaarsen · ad21154

L-\d -> L\d

2025-03-06 · by tomaarsen · 75b56ed

Adding `safetensors` variant of this model (#4)

2024-12-12 · by tomaarsen, SFconvertbot · 841d331

- Adding `safetensors` variant of this model (24b3ada7adf13a4c242643c6506bbc0f86575756)

Co-authored-by: Safetensors convertbot <[e-mail]>

Update README.md

2021-08-05 · by nreimers · e9ea268

upload flax model

2021-05-19 · by patrickvonplaten · 4a7941f

allow flax

2021-05-19 · by patrickvonplaten · 06a3f0e

update config

2021-04-20 · by nreimers · 2673f40

upload

2021-04-15 · by nreimers · 643fe93

initial commit

2021-04-15 · by system · 454e6d8