Socket
Socket
Sign inDemoInstall

fasttext-langdetect

Package Overview
Dependencies
0
Maintainers
1
Alerts
File Explorer

Install Socket

Protect your apps from supply chain attacks

Install

fasttext-langdetect

80x faster and 95% accurate language identification with Fasttext

    1.0.5

Maintainers
1

Readme

fasttext-langdetect

This library is a wrapper for the language detection model trained on fasttext by Facebook. For more information, please visit: https://fasttext.cc/docs/en/language-identification.html

Supported languages

af als am an ar arz as ast av az azb ba bar bcl be bg bh bn bo bpy br bs bxr ca cbk ce cebckb co cs cv cy da de diq dsb dty dv el eml en eo es et eu fa fi fr frr fy ga gd gl gn gom gu gv he hi hif hr hsb ht hu hy ia id ie ilo io is it ja jbo jv ka kk km kn ko krc ku kv kw ky la lb lez li lmo lo lrc lt lv mai mg mhr min mk ml mn mr mrj ms mt mwl my myv mzn nah nap nds ne new nl nn no oc or os pa pam pfl pl pms pnb ps pt qu rm ro ru rue sa sah sc scn sco sd sh si sk sl so sq sr su sv sw ta te tg th tk tl tr tt tyv ug uk ur uz vec vep vi vls vo wa war wuu xal xmf yi yo yue zh

Install

pip install fasttext-langdetect

Usage

detect method expects UTF-8 data. low_memory option enables getting predictions with the compressed version of the fasttext model by sacrificing the accuracy a bit.

from ftlangdetect import detect

result = detect(text="Bugün hava çok güzel", low_memory=False)
print(result)
> {'lang': 'tr', 'score': 1.00}

result = detect(text="Bugün hava çok güzel", low_memory=True)
print(result)
> {'lang': 'tr', 'score': 0.9982126951217651}

Benchmark

We benchmarked the fasttext model against cld2, langid, and langdetect on Wili-2018 dataset.

fasttextlangidlangdetectcld2
Average time (ms)0,1582733811,72661870512,446043170,028776978
139 langs - not weighted76,861,637,680,8
139 langs - pop weighted95,593,186,692,7
44 langs - not weighted93,389,281,691,5
44 langs - pop weighted96,694,889,493,4
  • pop weighted means recall for each language is multipled by its number of speakers.
  • 139 languages = all languages with ISO 639-1 2-letter code
  • 44 languages = top 44 languages spoken in the world
Recall per language
langcld2fasttextlangdetectlangid
Afrikaans0,940,9180,9920,966
Albanian0,9580,9660,9640,954
Amharic0,9760,98200,982
Arabic0,9940,9980,9980,996
Aragonese00,4300,788
Armenian0,9660,97200,968
Assamese0,9460,95600,14
Avar00,62600
Aymara0,596000
Azerbaijani0,970,98800,984
Bashkir0,970,9700
Basque0,9780,9900,962
Belarusian0,940,9700,964
Bengali0,8980,9220,9040,942
Bhojpuri0,7160,1500
Bokmål0,8520,9660,9760,95
Bosnian0,4220,10800,054
Breton0,9460,97400,976
Bulgarian0,8920,9640,9640,942
Burmese0,9980,99800
Catalan0,8820,950,930,928
Central Khmer0,8760,87800,876
Chechen00,9900
Chuvash00,9600
Cornish00,79200
Corsican0,880,01600
Croatian0,6880,8060,9820,932
Czech0,9780,9860,9840,982
Danish0,8860,9580,950,896
Dhivehi0,9960,99800
Dutch0,90,9780,9680,97
English0,99210,9980,986
Esperanto0,9360,97800,948
Estonian0,9180,9520,9480,932
Faroese0,912000,618
Finnish0,9880,9980,9980,994
French0,9460,9960,990,992
Galician0,890,91200,93
Georgian0,9740,97600,976
German0,9580,9840,9780,978
Guarani0,9680,72800
Gujarati0,9320,9320,930,932
Haitian Creole0,9880,53600,99
Hausa0,976000
Hebrew0,9940,9960,9980,998
Hindi0,9820,9840,9820,972
Hungarian0,960,9880,9680,986
Icelandic0,9840,99600,996
Ido00,7600
Igbo0,798000
Indonesian0,880,9460,9580,836
Interlingua0,270,68800
Interlingue0,1980,19200
Irish0,9680,97800,984
Italian0,8660,9480,9320,936
Japanese0,970,9860,980,986
Javanese00,86400,938
Kannada0,9980,9980,9980,998
Kazakh0,9780,99200,916
Kinyarwanda0,86000,44
Kirghiz0,9740,9900,408
Komi00,54400
Korean0,9860,990,9880,99
Kurdish00,97200,976
Lao0,840,84200,85
Latin0,7780,86400,854
Latvian0,980,9920,9920,99
Limburgan00,32400
Lingala0,85000
Lithuanian0,960,9760,9740,97
Luganda0,952000
Luxembourgish0,8640,89400,93
Macedonian0,880,9840,9820,974
Malagasy0,990,9900,988
Malay0,8960,58600,39
Malayalam0,9880,9880,9880,988
Maltese0,9620,96600,964
Manx0,9720,29400
Maori0,994000
Marathi0,9580,9660,9640,942
Modern Greek0,990,9920,990,992
Mongolian0,9640,99400,996
Navajo0000
Nepali (macrolanguage)0,960,980,9780,922
Northern Sami0000,866
Norwegian Nynorsk0,940,7900,796
Occitan0,660,4800,724
Oriya0,960,95800,96
Oromo0,956000
Ossetian00,93800
Panjabi0,9940,9940,9940,994
Persian0,9920,9980,9960,998
Polish0,9820,9980,9980,992
Portuguese0,9080,9560,9460,952
Pushto0,9380,92200,754
Quechua0,9260,80800,852
Romanian0,9320,9860,9840,984
Romansh0,9340,32800
Russian0,7280,9860,9840,988
Sanskrit0,9640,97600
Sardinian00,0100
Scottish Gaelic0,9640,94200
Serbian0,9420,94600,902
Serbo-Croatian00,40200
Shona0,844000
Sindhi0,9780,98200
Sinhala0,9620,96200,962
Slovak0,9640,9740,9820,97
Slovene0,8760,9660,9680,946
Somali0,9240,6960,9560
Spanish0,8940,9860,9760,98
Standard Chinese0,9460,9840,7460,978
Sundanese0,910,85400
Swahili (macrolanguage)0,9240,920,9380,934
Swedish0,8720,9940,9920,986
Tagalog0,9280,9720,9740,964
Tajik0,820,8500
Tamil0,9920,9920,9920,994
Tatar0,9780,98400
Telugu0,9580,9580,9580,96
Thai0,9880,9880,9880,988
Tibetan0,9860,99200
Tongan0,968000
Tswana0,928000
Turkish0,9680,9860,9820,976
Turkmen0,940,93600
Uighur0,9780,98600,964
Ukrainian0,970,9880,9860,986
Urdu0,860,9580,890,896
Uzbek0,9840,9900
Vietnamese0,9780,9860,9840,984
Volapük0,9940,98200,986
Walloon00,66400,98
Welsh0,980,9920,9920,984
Western Frisian0,8880,95600
Wolof0,926000
Xhosa0,928000,912
Yiddish0,9560,95800
Yoruba0,750,26200

References

[1] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of Tricks for Efficient Text Classification

@article{joulin2016bag,
  title={Bag of Tricks for Efficient Text Classification},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1607.01759},
  year={2016}
}

[2] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, T. Mikolov, FastText.zip: Compressing text classification models

@article{joulin2016fasttext,
  title={FastText.zip: Compressing text classification models},
  author={Joulin, Armand and Grave, Edouard and Bojanowski, Piotr and Douze, Matthijs and J{\'e}gou, H{\'e}rve and Mikolov, Tomas},
  journal={arXiv preprint arXiv:1612.03651},
  year={2016}
}

Keywords

FAQs


Did you know?

Socket installs a GitHub app to automatically flag issues on every pull request and report the health of your dependencies. Find out what is inside your node modules and prevent malicious activity before you update the dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc