@isdk/lingua

1. What does this library do?

Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.

2. Why does this library exist?

Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.

So far, other comprehensive open source libraries in the Rust ecosystem for this task are CLD2, Whatlang and Whichlang. Unfortunately, most of them have two major drawbacks:

1. Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, they do not provide adequate results.
2. The more languages take part in the decision process, the less accurate are the detection results.

Lingua aims at eliminating these problems. She nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. She draws on both rule-based and statistical methods but does not use any dictionaries of words. She does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

3. Which languages are supported?

Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, the following 75 languages are supported:

• A
  • Afrikaans
  • Albanian
  • Arabic
  • Armenian
  • Azerbaijani
• B
  • Basque
  • Belarusian
  • Bengali
  • Norwegian Bokmal
  • Bosnian
  • Bulgarian
• C
  • Catalan
  • Chinese
  • Croatian
  • Czech
• D
  • Danish
  • Dutch
• E
  • English
  • Esperanto
  • Estonian
• F
  • Finnish
  • French
• G
  • Ganda
  • Georgian
  • German
  • Greek
  • Gujarati
• H
  • Hebrew
  • Hindi
  • Hungarian
• I
  • Icelandic
  • Indonesian
  • Irish
  • Italian
• J
  • Japanese
• K
  • Kazakh
  • Korean
• L
  • Latin
  • Latvian
  • Lithuanian
• M
  • Macedonian
  • Malay
  • Maori
  • Marathi
  • Mongolian
• N
  • Norwegian Nynorsk
• P
  • Persian
  • Polish
  • Portuguese
  • Punjabi
• R
  • Romanian
  • Russian
• S
  • Serbian
  • Shona
  • Slovak
  • Slovene
  • Somali
  • Sotho
  • Spanish
  • Swahili
  • Swedish
• T
  • Tagalog
  • Tamil
  • Telugu
  • Thai
  • Tsonga
  • Tswana
  • Turkish
• U
  • Ukrainian
  • Urdu
• V
  • Vietnamese
• W
  • Welsh
• X
  • Xhosa
• Y
  • Yoruba
• Z
  • Zulu

4. How accurate is it?

Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:

1. a list of single words with a minimum length of 5 characters
2. a list of word pairs with a minimum length of 10 characters
3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the Wortschatz corpora offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua, CLD2, Whatlang and Whichlang running over the data of Lingua's supported 75 languages. Languages that are not supported by the other libraries are simply ignored for the respective library during the detection process.

Each of the following sections contains four plots. The bar plots show the detailed accuracy results for each supported language. The box plots illustrate the distributions of the accuracy values for each classifier. The boxes themselves represent the areas which the middle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the median of the distributions.

The first two plots in each section show the results for all supported languages in each classifier, respectively. The last two plots are restricted to the common subset of currently 16 languages that is supported by all compared classifiers. This distinction makes sense because the first box plot creates the impression that Whichlang is the most accurate classifier, but it is not. Whichlang supports only 16 languages whereas Lingua supports 75 languages. For the second box plot, the supported languages in Whatlang and Lingua have been restricted to those 16 languages supported by Whichlang. This provides for a more accurate comparison and shows that overall, Lingua is the most accurate language detection library in this comparison.

4.1 Single word detection

4.1.1 All languages

Bar plot

4.1.2 Common languages

Bar plot

4.2 Word pair detection

4.2.1 All languages

Bar plot

4.2.2 Common languages

Bar plot

4.3 Sentence detection

4.3.1 All languages

Bar plot

4.3.2 Common languages

Bar plot

4.4 Average detection

4.4.1 All languages

Bar plot

4.4.2 Common languages

Bar plot

4.5 Mean, median and standard deviation

The tables below show detailed statistics for each language and classifier including mean, median and standard deviation.

4.5.1 All languages

Open table

Language	Average	Single Words	Word Pairs	Sentences
	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2
Afrikaans	78	64	-	51	55	58	37	-	22	12	80	62	-	41	56	96	93	-	92	96
Albanian	87	79	-	-	64	68	54	-	-	18	94	86	-	-	76	99	98	-	-	99
Arabic	98	94	100	93	66	96	88	100	88	19	99	95	100	92	81	99	99	100	99	99
Armenian	100	100	-	-	100	100	100	-	-	100	100	100	-	-	100	100	100	-	-	100
Azerbaijani	89	82	-	79	71	77	71	-	61	34	92	78	-	81	81	99	96	-	97	99
Basque	83	74	-	-	61	71	55	-	-	22	87	76	-	-	69	92	91	-	-	91
Belarusian	96	91	-	91	75	91	80	-	81	42	99	95	-	94	87	99	99	-	99	98
Bengali	100	100	-	99	62	100	100	-	100	19	100	100	-	100	69	100	100	-	99	99
Bokmal	58	49	-	49	-	38	27	-	24	-	58	47	-	44	-	76	74	-	80	-
Bosnian	34	29	-	-	18	29	22	-	-	4	34	28	-	-	15	40	36	-	-	36
Bulgarian	86	77	-	69	65	70	56	-	44	31	91	80	-	67	72	99	96	-	96	92
Catalan	70	58	-	51	37	50	33	-	29	4	73	60	-	45	29	86	81	-	80	79
Chinese	100	100	97	100	33	100	100	93	100	-	100	100	98	100	2	100	100	100	100	98
Croatian	72	59	-	61	51	53	36	-	34	33	74	57	-	54	46	90	85	-	94	72
Czech	80	70	-	63	73	65	54	-	42	50	84	71	-	66	79	91	87	-	82	90
Danish	81	70	-	53	59	61	45	-	31	26	83	70	-	45	56	97	95	-	84	94
Dutch	77	63	73	47	47	55	35	53	22	10	80	61	70	37	41	96	94	98	82	90
English	80	62	66	49	55	54	29	34	17	11	88	62	65	35	55	99	96	98	94	99
Esperanto	83	65	-	58	50	67	43	-	32	7	85	60	-	53	45	98	92	-	90	97
Estonian	91	83	-	67	65	79	62	-	43	23	96	87	-	62	73	99	99	-	96	99
Finnish	96	90	-	76	77	90	77	-	53	43	98	94	-	78	89	99	99	-	98	99
French	89	77	68	63	51	74	51	36	35	12	94	82	70	59	47	99	97	99	97	93
Ganda	91	83	-	-	61	79	64	-	-	23	95	86	-	-	62	100	99	-	-	98
Georgian	99	99	-	99	100	100	100	-	100	100	100	100	-	100	100	99	99	-	99	100
German	89	79	79	69	63	73	56	55	44	26	94	83	82	66	66	99	99	99	98	98
Greek	99	99	-	99	100	100	100	-	100	100	100	100	-	100	100	99	99	-	99	100
Gujarati	99	99	-	99	99	99	99	-	100	99	100	100	-	100	100	100	100	-	99	100
Hebrew	99	99	-	95	-	100	100	-	92	-	100	100	-	95	-	99	99	-	99	-
Hindi	73	32	100	68	76	60	11	100	54	55	64	19	100	54	75	94	67	100	96	99
Hungarian	94	90	-	72	75	86	77	-	50	40	97	93	-	70	85	100	99	-	98	99
Icelandic	93	87	-	-	65	82	71	-	-	25	96	92	-	-	73	99	99	-	-	99
Indonesian	60	47	-	69	62	39	25	-	42	36	60	45	-	69	62	82	71	-	95	88
Irish	90	85	-	-	66	81	70	-	-	29	94	90	-	-	77	95	94	-	-	92
Italian	86	71	85	56	44	69	41	68	25	6	91	73	87	48	32	99	98	99	96	93
Japanese	100	100	99	99	33	100	100	98	98	-	100	100	100	100	-	100	100	99	100	100
Kazakh	95	94	-	-	76	89	87	-	-	43	97	94	-	-	87	99	99	-	-	99
Korean	99	99	99	99	99	100	100	100	100	100	100	100	100	100	100	99	99	99	99	99
Latin	87	73	-	55	46	72	49	-	36	8	92	75	-	51	42	97	94	-	79	88
Latvian	93	87	-	85	72	84	74	-	70	33	96	90	-	86	84	98	96	-	97	98
Lithuanian	94	87	-	79	70	86	75	-	59	29	97	88	-	81	81	99	98	-	97	99
Macedonian	83	72	-	66	60	65	51	-	43	27	86	70	-	60	69	98	95	-	96	83
Malay	30	30	-	-	17	26	21	-	-	9	38	35	-	-	21	28	35	-	-	23
Maori	92	83	-	-	60	84	63	-	-	12	92	87	-	-	71	99	97	-	-	98
Marathi	84	38	-	78	83	73	15	-	59	65	84	29	-	80	86	96	71	-	94	98
Mongolian	96	95	-	-	78	92	87	-	-	43	98	98	-	-	92	99	99	-	-	99
Nynorsk	65	51	-	-	54	40	25	-	-	18	65	49	-	-	50	90	81	-	-	93
Persian	90	80	-	72	61	77	61	-	49	12	93	80	-	68	72	99	97	-	99	99
Polish	94	89	-	81	74	85	76	-	63	37	98	93	-	81	86	99	99	-	99	99
Portuguese	81	68	70	60	53	59	41	42	30	19	85	69	70	52	47	98	95	99	97	94
Punjabi	99	99	-	99	100	100	100	-	100	100	100	100	-	100	100	99	99	-	99	100
Romanian	86	72	-	66	53	68	49	-	45	11	91	73	-	62	53	99	94	-	90	96
Russian	89	78	100	67	60	76	59	100	51	26	94	83	100	66	67	97	92	100	84	86
Serbian	87	77	-	66	68	73	61	-	46	29	90	79	-	63	77	99	91	-	89	99
Shona	91	80	-	76	64	77	56	-	55	23	95	86	-	76	71	100	99	-	98	99
Slovak	84	75	-	64	71	64	49	-	40	37	90	78	-	60	75	98	97	-	93	99
Slovene	82	66	-	52	47	61	39	-	29	8	86	67	-	43	42	98	93	-	83	92
Somali	92	84	-	-	69	81	64	-	-	26	95	90	-	-	83	99	99	-	-	99
Sotho	85	71	-	-	53	66	42	-	-	12	90	74	-	-	54	99	97	-	-	95
Spanish	69	56	62	47	43	43	25	35	18	12	68	48	54	32	33	97	93	98	91	84
Swahili	80	69	-	-	57	60	43	-	-	15	84	68	-	-	58	98	96	-	-	97
Swedish	83	72	71	61	53	64	45	47	34	14	88	75	70	57	52	98	94	97	92	93
Tagalog	77	66	-	-	50	52	35	-	-	9	83	66	-	-	44	98	96	-	-	96
Tamil	100	100	-	100	100	100	100	-	100	100	100	100	-	100	100	100	100	-	100	100
Telugu	100	100	-	99	100	100	100	-	100	100	100	100	-	100	100	100	100	-	99	100
Thai	99	99	-	99	99	100	100	-	100	100	100	100	-	100	100	99	99	-	99	99
Tsonga	84	72	-	-	61	66	46	-	-	19	89	73	-	-	67	98	96	-	-	97
Tswana	84	71	-	-	56	65	44	-	-	17	88	73	-	-	57	98	96	-	-	94
Turkish	93	87	92	63	66	83	70	83	39	30	97	91	94	59	71	99	99	99	91	97
Ukrainian	92	86	-	85	77	84	74	-	69	45	97	91	-	88	87	95	92	-	98	98
Urdu	90	78	-	68	61	80	64	-	45	8	94	78	-	63	75	95	94	-	95	99
Vietnamese	90	87	85	91	63	78	75	64	75	-	94	87	91	97	89	99	98	99	99	99
Welsh	91	82	-	-	72	78	60	-	-	34	95	87	-	-	85	99	98	-	-	98
Xhosa	82	68	-	-	71	63	44	-	-	45	85	67	-	-	70	98	93	-	-	97
Yoruba	74	62	-	-	36	50	33	-	-	1	76	60	-	-	22	96	92	-	-	87
Zulu	80	70	-	76	54	62	45	-	53	18	83	71	-	77	51	97	94	-	99	93
Mean	86	77	84	74	64	74	60	69	57	34	89	78	84	71	68	96	93	99	94	94
Median	89.0	78.0	85.0	69.0	63.0	74.0	56.0	66.0	49.5	26.0	93.0	80.0	89.0	67.5	71.0	99.0	96.0	99.0	97.0	98.0
Standard Deviation	13.17	17.39	14.21	17.27	18.54	18.54	25.09	26.6	27.23	28.81	13.25	19.12	15.66	21.07	22.74	11.06	11.84	0.85	6.04	12.16

4.5.2 Common languages

Open table

Language	Average	Single Words	Word Pairs	Sentences
	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Whichlang	Whatlang	CLD2
Arabic	100	100	100	100	66	100	100	100	100	19	100	100	100	100	81	100	100	100	100	99
Chinese	100	100	97	100	33	100	100	93	100	-	100	100	98	100	2	100	100	100	100	98
Dutch	87	80	73	63	47	73	60	53	39	10	89	81	70	54	41	99	98	98	96	90
English	87	76	66	57	55	70	52	34	29	11	93	77	65	47	55	99	98	98	95	99
French	92	84	68	74	51	82	65	36	50	12	96	88	70	73	47	99	98	99	98	93
German	92	86	79	76	63	82	70	55	54	26	95	89	82	75	66	99	99	99	98	98
Hindi	99	99	100	99	76	100	100	100	100	55	100	100	100	100	75	99	99	100	99	99
Italian	92	82	85	68	44	82	63	68	45	6	95	84	87	63	32	99	99	99	97	93
Japanese	100	100	99	99	33	100	100	98	98	-	100	100	100	100	-	100	100	99	100	100
Korean	99	99	99	99	99	100	100	100	100	100	100	100	100	100	100	99	99	99	99	99
Portuguese	85	76	70	67	53	69	55	42	42	19	88	77	70	62	47	99	97	99	98	94
Russian	100	100	100	100	60	100	100	100	100	26	100	100	100	100	67	100	100	100	100	86
Spanish	78	69	62	54	43	59	46	35	29	12	78	65	54	41	33	98	95	98	93	84
Swedish	93	86	71	74	53	83	69	47	51	14	96	90	70	74	52	99	98	97	97	93
Turkish	97	95	92	81	66	94	89	83	66	30	99	97	94	78	71	100	99	99	98	97
Vietnamese	94	91	85	92	63	86	83	64	79	-	96	92	91	98	89	99	99	99	99	99
Mean	93	89	84	81	57	86	78	69	68	26	95	90	84	79	57	99	99	99	98	95
Median	93.5	88.5	85.0	78.5	54.0	84.5	76.5	66.0	60.0	19.0	96.0	91.0	89.0	76.5	55.0	99.0	99.0	99.0	98.0	97.5
Standard Deviation	6.56	10.44	14.21	16.91	16.44	13.51	20.23	26.6	28.33	25.57	6.03	10.76	15.66	21.17	25.16	0.58	1.31	0.85	1.98	4.95

5. How fast is it?

Various benchmarks for all classifiers can be run via:

cargo bench --features benchmark

The benchmarks measure the processing time for classifying 2,000 sentences available in the common 16 languages that are supported by each language detector. The results below have been produced on an iMac 3.6 Ghz 8-Core Intel Core i9 with 40 GB RAM. Whichlang has the shortest processing time, Lingua the longest.

Detector	Single Thread	Multiple Threads
Whichlang	2.046 ms	351.03 µs
CLD 2	8.923 ms	2.05 ms
Whatlang (common languages)	47.74 ms	5.61 ms
Whatlang (all languages)	113.08 ms	12.99 ms
Lingua (low accuracy mode, common languages)	180.54 ms	24.55 ms
Lingua (high accuracy mode, common languages)	333.31 ms	37.35 ms
Lingua (low accuracy mode, all languages)	373.15 ms	48.18 ms
Lingua (high accuracy mode, all languages)	622.00 ms	96.65 ms

The accuracy reporter script measures the time each language detector needs to classify 3000 input texts for each of the supported 75 languages.

Detector	Time
Whichlang	0.13 sec
CLD 2	1.67 sec
Lingua (low accuracy mode, multi-threaded)	3.11 sec
Whatlang	9.82 sec
Lingua (high accuracy mode, multi-threaded)	10.36 sec
Lingua (low accuracy mode, single-threaded)	21.06 sec
Lingua (high accuracy mode, single-threaded)	57.15 sec

6. Why is it better than other libraries?

Every language detector uses a probabilistic n-gram model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

7. How to reproduce the accuracy results?

If you want to reproduce the accuracy results above, you can generate the test reports yourself for all classifiers and all languages by doing:

cargo run --release --bin accuracy_reports --features accuracy-reports

It is important to use the --release flag here because loading the language models in debug mode takes too much time. For each detector and language, a test report file is then written into /accuracy-reports, to be found next to the src directory. As an example, here is the current output of the Lingua German report:

##### German #####

>>> Accuracy on average: 89.23%

>> Detection of 1000 single words (average length: 9 chars)
Accuracy: 73.9%
Erroneously classified as Dutch: 2.3%, Danish: 2.1%, English: 2%, Latin: 1.9%, Bokmal: 1.6%, Basque: 1.2%, Esperanto: 1.2%, French: 1.2%, Italian: 1.2%, Swedish: 1%, Afrikaans: 0.8%, Tsonga: 0.7%, Nynorsk: 0.6%, Spanish: 0.6%, Yoruba: 0.6%, Finnish: 0.5%, Sotho: 0.5%, Welsh: 0.5%, Estonian: 0.4%, Irish: 0.4%, Polish: 0.4%, Swahili: 0.4%, Tagalog: 0.4%, Tswana: 0.4%, Bosnian: 0.3%, Icelandic: 0.3%, Romanian: 0.3%, Albanian: 0.2%, Catalan: 0.2%, Croatian: 0.2%, Indonesian: 0.2%, Lithuanian: 0.2%, Maori: 0.2%, Turkish: 0.2%, Xhosa: 0.2%, Zulu: 0.2%, Latvian: 0.1%, Malay: 0.1%, Slovak: 0.1%, Slovene: 0.1%, Somali: 0.1%

>> Detection of 1000 word pairs (average length: 18 chars)
Accuracy: 94.1%
Erroneously classified as Dutch: 0.9%, Latin: 0.8%, English: 0.7%, Swedish: 0.6%, Danish: 0.5%, French: 0.4%, Bokmal: 0.3%, Irish: 0.2%, Tagalog: 0.2%, Afrikaans: 0.1%, Esperanto: 0.1%, Estonian: 0.1%, Finnish: 0.1%, Italian: 0.1%, Maori: 0.1%, Nynorsk: 0.1%, Somali: 0.1%, Swahili: 0.1%, Tsonga: 0.1%, Turkish: 0.1%, Welsh: 0.1%, Zulu: 0.1%

>> Detection of 1000 sentences (average length: 111 chars)
Accuracy: 99.7%
Erroneously classified as Dutch: 0.2%, Latin: 0.1%

8. How to add it to your project?

Add Lingua to your Cargo.toml file like so:

[dependencies]
lingua = "1.6.2"

By default, this will download the language model dependencies for all 75 supported languages, a total of approximately 90 MB. If your bandwidth or hard drive space is limited, or you simply do not need all languages, you can specify a subset of the language models to be downloaded as separate features in your Cargo.toml:

[dependencies]
lingua = { version = "1.6.2", default-features = false, features = ["french", "italian", "spanish"] }

9. How to build?

In order to build the source code yourself, you need the stable Rust toolchain installed on your machine so that cargo, the Rust package manager is available.

git clone https://github.com/pemistahl/lingua-rs.git
cd lingua-rs
cargo build

The source code is accompanied by an extensive unit test suite. To run them, simply say:

cargo test

With the help of PyO3 and Maturin, the library has been compiled to a Python extension module so that it can be used within any Python software as well. It is available in the Python Package Index and can be installed with:

pip install lingua-language-detector

To build the Python extension module yourself, create a virtual environment and install Maturin.

python -m venv .venv
source .venv/bin/activate
pip install maturin
maturin build

10. How to use?

10.1 Basic usage

use lingua::{Language, LanguageDetector, LanguageDetectorBuilder};
use lingua::Language::{English, French, German, Spanish};

let languages = vec![English, French, German, Spanish];
let detector: LanguageDetector = LanguageDetectorBuilder::from_languages(&languages).build();
let detected_language: Option<Language> = detector.detect_language_of("languages are awesome");

assert_eq!(detected_language, Some(English));

10.2 Minimum relative distance

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word prologue, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated in the following way:

use lingua::LanguageDetectorBuilder;
use lingua::Language::{English, French, German, Spanish};

let detector = LanguageDetectorBuilder::from_languages(&[English, French, German, Spanish])
    .with_minimum_relative_distance(0.9)
    .build();
let detected_language = detector.detect_language_of("languages are awesome");

assert_eq!(detected_language, None);

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high. Otherwise None will be returned most of the time as in the example above. This is the return value for cases where language detection is not reliably possible.

10.3 Confidence values

Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? These questions can be answered as well:

use lingua::Language::{English, French, German, Spanish};
use lingua::LanguageDetectorBuilder;

fn main() { 
    let languages = vec![English, French, German, Spanish];
    let detector = LanguageDetectorBuilder::from_languages(&languages).build();
    let confidence_values = detector
        .compute_language_confidence_values("languages are awesome")
        .into_iter()
        .map(|(language, confidence)| (language, (confidence * 100.0).round() / 100.0))
        .collect::<Vec<_>>();

    assert_eq!(
        confidence_values,
        vec![
            (English, 0.93),
            (French, 0.04),
            (German, 0.02),
            (Spanish, 0.01)
        ]
    );
}

In the example above, a vector of two-element tuples is returned containing all possible languages sorted by their confidence value in descending order. Each value is a probability between 0.0 and 1.0. The probabilities of all languages will sum to 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned for this language. The other languages will receive a value of 0.0.

There is also a method for returning the confidence value for one specific language only:

use lingua::Language::{English, French, German, Spanish};
use lingua::LanguageDetectorBuilder;

fn main() {
    let languages = vec![English, French, German, Spanish];
    let detector = LanguageDetectorBuilder::from_languages(&languages).build();
    let confidence = detector.compute_language_confidence("languages are awesome", French);
    let rounded_confidence = (confidence * 100.0).round() / 100.0;

    assert_eq!(rounded_confidence, 0.04);
}

The value that this method computes is a number between 0.0 and 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned. If the given language is not supported by this detector instance, the value 0.0 will always be returned.

10.4 Eager loading versus lazy loading

By default, Lingua uses lazy-loading to load only those language models on demand which are considered relevant by the rule-based filter engine. For web services, for instance, it is rather beneficial to preload all language models into memory to avoid unexpected latency while waiting for the service response. If you want to enable the eager-loading mode, you can do it like this:

LanguageDetectorBuilder::from_all_languages().with_preloaded_language_models().build();

Multiple instances of LanguageDetector share the same language models in memory which are accessed asynchronously by the instances.

10.5 Low accuracy mode versus high accuracy mode

Lingua's high detection accuracy comes at the cost of being noticeably slower than other language detectors. The large language models also consume significant amounts of memory. These requirements might not be feasible for systems running low on resources. If you want to classify mostly long texts or need to save resources, you can enable a low accuracy mode that loads only a small subset of the language models into memory:

LanguageDetectorBuilder::from_all_languages().with_low_accuracy_mode().build();

The downside of this approach is that detection accuracy for short texts consisting of less than 120 characters will drop significantly. However, detection accuracy for texts which are longer than 120 characters will remain mostly unaffected.

In high accuracy mode (the default), the language detector consumes approximately 970 MB of memory if all language models are loaded. In low accuracy mode, memory consumption is reduced to approximately 72 MB. The goal is to further reduce memory consumption in later releases.

An alternative for a smaller memory footprint and faster performance is to reduce the set of languages when building the language detector. In most cases, it is not advisable to build the detector from all supported languages. When you have knowledge about the texts you want to classify you can almost always rule out certain languages as impossible or unlikely to occur.

10.6 Detection of multiple languages in mixed-language texts

In contrast to most other language detectors, Lingua is able to detect multiple languages in mixed-language texts. This feature can yield quite reasonable results, but it is still in an experimental state and therefore the detection result is highly dependent on the input text. It works best in high-accuracy mode with multiple long words for each language. The shorter the phrases and their words are, the less accurate are the results. Reducing the set of languages when building the language detector can also improve accuracy for this task if the languages occurring in the text are equal to the languages supported by the respective language detector instance.

use lingua::DetectionResult;
use lingua::Language::{English, French, German};
use lingua::LanguageDetectorBuilder;

fn main() {
    let languages = vec![English, French, German];
    let detector = LanguageDetectorBuilder::from_languages(&languages).build();
    let sentence = "Parlez-vous français? \
        Ich spreche Französisch nur ein bisschen. \
        A little bit is better than nothing.";

    let results: Vec<DetectionResult> = detector.detect_multiple_languages_of(sentence);

    if let [first, second, third] = &results[..] {
        assert_eq!(first.language(), French);
        assert_eq!(
            &sentence[first.start_index()..first.end_index()],
            "Parlez-vous français? "
        );

        assert_eq!(second.language(), German);
        assert_eq!(
            &sentence[second.start_index()..second.end_index()],
            "Ich spreche Französisch nur ein bisschen. "
        );

        assert_eq!(third.language(), English);
        assert_eq!(
            &sentence[third.start_index()..third.end_index()],
            "A little bit is better than nothing."
        );
    }
}

In the example above, a vector of DetectionResult is returned. Each entry in the vector describes a contiguous single-language text section, providing start and end indices of the respective substring.

10.7 Single-threaded versus multi-threaded language detection

The LanguageDetector methods explained above all operate in a single thread. If you want to classify a very large set of texts, you will probably want to use all available CPU cores efficiently in multiple threads for maximum performance.

Every single-threaded method has a multi-threaded equivalent that accepts a list of texts and returns a list of results.

Single-threaded	Multi-threaded
detect_language_of	detect_languages_in_parallel_of
detect_multiple_languages_of	detect_multiple_languages_in_parallel_of
compute_language_confidence_values	compute_language_confidence_values_in_parallel
compute_language_confidence	compute_language_confidence_in_parallel

10.8 Methods to build the LanguageDetector

There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance (what a surprise :-). The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages:

use lingua::{LanguageDetectorBuilder, Language, IsoCode639_1, IsoCode639_3};

// Include all languages available in the library.
LanguageDetectorBuilder::from_all_languages();

// Include only languages that are not yet extinct (= currently excludes Latin).
LanguageDetectorBuilder::from_all_spoken_languages();

// Include only languages written with Cyrillic script.
LanguageDetectorBuilder::from_all_languages_with_cyrillic_script();

// Exclude only the Spanish language from the decision algorithm.
LanguageDetectorBuilder::from_all_languages_without(&[Language::Spanish]);

// Only decide between English and German.
LanguageDetectorBuilder::from_languages(&[Language::English, Language::German]);

// Select languages by ISO 639-1 code.
LanguageDetectorBuilder::from_iso_codes_639_1(&[IsoCode639_1::EN, IsoCode639_1::DE]);

// Select languages by ISO 639-3 code.
LanguageDetectorBuilder::from_iso_codes_639_3(&[IsoCode639_3::ENG, IsoCode639_3::DEU]);

11. WebAssembly support

This library can be compiled to WebAssembly (WASM) which allows to use Lingua in any JavaScript-based project, be it in the browser or in the back end running on Node.js.

The easiest way to compile is to use wasm-pack. After the installation, you can, for instance, build the library with the web target so that it can be directly used in the browser:

wasm-pack build --target web

By default, all 75 supported languages are included in the compiled wasm file which has a size of 74 MB, approximately. If you only need a subset of certain languages, you can tell wasm-pack which ones to include:

wasm-pack build --target web -- --no-default-features --features "french,italian,spanish"

This creates a directory named pkg on the top-level of this repository, containing the compiled wasm files and JavaScript and TypeScript bindings. In an HTML file, you can then call Lingua like the following, for instance:

<script type="module">
    import init, { LanguageDetectorBuilder } from './pkg/lingua.js';

    init().then(_ => {
        const detector = LanguageDetectorBuilder.fromAllLanguages().build();
        console.log(detector.computeLanguageConfidenceValues("languages are awesome"));
    });
</script>

There are also some integration tests available for Node.js and all major browsers. To run them, simply say:

wasm-pack test --node --headless --chrome --firefox --safari

The output of wasm-pack will be hosted in a separate repository which allows to add further JavaScript-related configuration, tests and documentation. Lingua will then be added to the npm registry as well, allowing for an easy download and installation within every JavaScript or TypeScript project.

12. What's next for version 1.7.0?

Take a look at the planned issues.

13. Contributions

• Josh Rotenberg has written a wrapper for using Lingua with the Elixir programming language.

• Simon Liang has written a wrapper for using Lingua with NodeJS.

Any contributions to Lingua are very much appreciated. Please read the instructions in CONTRIBUTING.md for how to add new languages to the library.