@genesys225/fast-text-language-detection

Fast-Text Language Detection

In a search for the best option for predicting a language from text which didn't require a large machine learning model, it appeared that fast-text, created by FaceBook, was the best option (https://towardsdatascience.com/benchmarking-language-detection-for-nlp-8250ea8b67c).

Installation

npm i --save @smodin/fast-text-language-detection

Note: This will install the fast-text model by facebook which is about 150MB. You also need python installed, if you're running an alipine docker see how to easily do this here

Usage

Prediction

Testing

;(async () => {
  const LanguageDetection = require('@smodin/fast-text-language-detection')
  const lid = new LanguageDetection()

  console.log(await lid.predict('FastText-LID provides a great language identification'))
  console.log(await lid.predict('FastText-LID bietet eine hervorragende Sprachidentifikation'))
  console.log(await lid.predict('FastText-LID fornisce un ottimo linguaggio di identificazione'))
  console.log(await lid.predict('FastText-LID fournit une excellente identification de la langue'))
  console.log(await lid.predict('FastText-LID proporciona una gran identificación de idioma'))
  console.log(await lid.predict('FastText-LID обеспечивает отличную идентификацию языка'))
  console.log(await lid.predict('FastText-LID提供了很好的語言識別'))
})()

The second argument is the number of returned responses, i.e. lid.predict(text, 10) will return an array of 10 results

Output

[ { lang: 'en', prob: 0.6313226222991943, isReliableLanguage: true } ]
[ { lang: 'de', prob: 0.9137917160987854, isReliableLanguage: true } ]
[ { lang: 'it', prob: 0.974501371383667, isReliableLanguage: true } ]
[ { lang: 'fr', prob: 0.7358829379081726, isReliableLanguage: true } ]
[ { lang: 'es', prob: 0.9211937189102173, isReliableLanguage: true } ]
[ { lang: 'ru', prob: 0.9899846911430359, isReliableLanguage: true } ]
[ { lang: 'zh', prob: 0.8515647649765015, isReliableLanguage: true } ]

isReliableLanguage is true if there were 10 + test results and accuracy was 95% or more

Other Helpers

const LanguageDetection = require('@smodin/fast-text-language-detection')
const lid = new LanguageDetection()
const languageIsoCodes = lid.languageIsoCodes // ['af', 'als', 'am', 'an', 'ar', ...]

Similar Libaries

FastText has been used and implemented in other computer languages.

• Python[https://github.com/indix/whatthelang]

Reference Documents

• FastText model 176: https://fasttext.cc/docs/en/language-identification.html

Accuracy from Benchmark Testing

Long Input (30 to 250 characters)

Translated sentence data was obtained from tatoeba.org. Additional meta data can be found in benchmark-testing/results/RESULTS_with_metadata.csv.

Testing the 550k sentences of 30 - 250 characters took less than 30 seconds (personal macbook Pro).

Language (101)	Symbol (alternates)	Count (558260)	Accuracy (30 - 250 chars)	Mislabels	False Positives
English	en	22428	1		120
Greek	el	12039	1		0
Hebrew	he	8616	1		0
Japanese	ja	2169	1		0
Georgian	ka	1973	1		0
Bengali	bn	1164	1		131
Thai	th	572	1		0
Mandarin Chinese	zh	568	1		0
Malayalam	ml	517	1		0
Korean	ko	482	1		7
Burmese	my	216	1		0
Tamil	ta	205	1		0
Kannada	kn	118	1		1
Telugu	te	102	1		0
Punjabi (Eastern)	pa	88	1		0
Lao	lo	70	1		0
Gujarati	gu	57	1		0
Tibetan	bo	20	1		0
Divehi, Dhivehi, Maldivian	dv	15	1		0
Sinhala	si	9	1		0
Amharic	am	3	1		0
German	de	22014	0.9998637230853094	en	64
Polish	pl	17768	0.999718595227375	en,eo,de,ro	88
Russian	ru	17329	0.9997114663281205	bg,kk,uk,mk	241
Hungarian	hu	17942	0.9996655891204994	tr,br,it,de,en	43
Hindi	hi	5362	0.999627004848937	mr	0
Vietnamese	vi	13000	0.9996153846153846	eo,hu,fr	9
Turkish	tr	19919	0.9995983734123199	eo,en,it,fr,nds	1092
Esperanto	eo	17841	0.999551594641556	it,es,pt,fr,ceb	13
French	fr	23076	0.999523314265904	en,es,it,ru	238
Marathi	mr	10461	0.9995220342223496	hi	2
Uyghur	ug	3692	0.9991874322860238	ba,ru,hu	0
Finnish	fi	17406	0.9990807767436516	it,et,en,hr,de	37
Italian	it	18326	0.9989632216522972	es,de,fr,en,la	2207
Spanish	es	18227	0.998134635430954	pt,it,io,ca,ia	3476
Armenian	hy	518	0.9980694980694981	de	0
Arabic	ar	8761	0.9978312977970552	arz,fa,es,mzn,en	0
Ukrainian	uk	14285	0.9963598179908996	ru,sr	133
Macedonian	mk	14465	0.9959903214656066	bg,sr,ru	93
Dutch	nl	19626	0.9934780393355752	en,af,de,nds,fr	382
Lithuanian	lt	13835	0.9933501987712324	fi,pl,eo,pt,sr	20
Portuguese	pt	20174	0.9933082184990581	es,gl,it,en,fr	1149
Khmer	km	379	0.9920844327176781	az,et	0
Urdu	ur	963	0.9906542056074766	pnb,fa,ro,en	9
Czech	cs	10863	0.9898738838258307	sk,pl,hu,sl,en	1
Swedish	sv	12188	0.9886773875943551	no,da,en,fi,id	174
Romanian	ro	13560	0.9886430678466077	es,fr,it,en,pt	133
Bulgarian	bg	11144	0.9869885139985642	mk,ru,uk,sr	2
Ossetian	os	59	0.9830508474576272	ru	0
Icelandic	is	6364	0.9803582652419862	et,no,da,hu,cs	4
Kazakh	kk	2232	0.9802867383512545	ru,tr,tt,uk,ky	4
Tagalog	tl	10351	0.9737223456670853	ceb,en,id,es,war	21
Tatar	tt	8178	0.9680851063829787	az,tr,ru,fi,kk	13
Basque	eu	2999	0.9676558852950984	it,nl,id,en,io	14
Tajik	tg	30	0.9666666666666667	ru	0
Belarusian	be	6253	0.9625779625779626	uk,ru,pl,bg,sr	0
Latvian	lv	1243	0.9597747385358005	lt,hr,sr,fi,eo	4
Chuvash	cv	460	0.9543478260869566	ru,uk,ba,sr	0
Breton	br	2451	0.9543043655650755	fr,nl,eu,de,pt	0
Bashkir	ba	120	0.95	tt,av	0
Indonesian	id	9372	0.949637217242851	ms,it,en,eo,tr	16
Danish	da	15299	0.948035819334597	no,sv,de,en,nn	2
Estonian	et	1227	0.9356153219233904	fi,en,hu,it,nl	5
Latin	la	11437	0.9206085511934948	fr,it,en,es,pt	292
Irish	ga	867	0.9065743944636678	en,gd,ca,kv,cs	14
Scottish Gaelic	gd	542	0.8966789667896679	en,ga,de,fr,pam	2
Welsh	cy	619	0.8917609046849758	es,en,la,kw,de	8
Catalan	ca	4725	0.8833862433862434	es,pt,fr,it,ro	0
Kyrgyz	ky	66	0.8787878787878788	ru,kk	4
Cornish	kw	426	0.8779342723004695	en,cy,de,br,sq	1
Assamese	as	960	0.8635416666666667	bn	0
Volapük	vo	806	0.8511166253101737	id,de,fi,en,eo	15
Serbian	sr	13494	0.8489699125537276	hr,sh,mk,bs,sl	1050
Slovak	sk	4370	0.8263157894736842	cs,pl,sl,no,sr	45
Maltese	mt	52	0.8076923076923077	es,cs,pt,sr,eo	7
Norwegian Nynorsk	nn (no)	657	0.7990867579908676	da,sv,de,es,fi	29
Afrikaans	af	1632	0.7879901960784313	nl,en,fr,de,nds	0
Occitan	oc	2861	0.7679133170220203	ca,es,fr,pt,it	27
Interlingua	ia	18782	0.7500798636992866	es,it,fr,la,pt	82
Sanskrit	sa	11	0.7272727272727273	hi,ne	0
Chechen	ce	7	0.7142857142857143	mn,ru	0
Slovenian	sl	372	0.6774193548387096	sr,hr,bs,pl,eo	62
Frisian	fy	107	0.6635514018691588	nl,en,de,af,fr	8
Javanese	jv	260	0.6461538461538462	id,en,ms,ko,su	5
Yoruba	yo	5	0.6	sk,rm	1
Luxembourgish	lb	217	0.5944700460829493	de,nds,sv,fr,nl	3
Galician	gl	2618	0.5790679908326967	pt,es,it,fr,ca	8
Turkmen	tk	3793	0.5710519377801213	tr,uz,en,et,io	0
Croatian	hr	2222	0.5333033303330333	sr,sh,bs,sl,pl	45
Aragonese	an	4	0.5	es	0
Ido	io	2905	0.48055077452667816	eo,es,it,pt,tr	7
Interlingue	ie	2007	0.4718485301444943	es,it,fr,en,ia	7
Limburgan, Limburger, Limburgish	li	3	0.3333333333333333	de	1
Walloon	wa	16	0.3125	fr,pt,tl,oc,en	1
Somali	so	32	0.21875	fi,eo,cy,en,az	1
Corsican	co	5	0.2	it,fr	0
Sundanese	su	11	0.18181818181818182	id,ms,es	19
Haitian Creole	ht	15	0.06666666666666667	br,fr,su,diq,no	3
Romansh	rm	16	0.0625	it,fr,en,tl,qu	3
Bosnian	bs	139	0.03597122302158273	sr,hr,sh,pl,sl	0
Manx	gv	6	0	cy,fr,nl,et,en	0

Short Form (10 to 40 characters)

As a test of accuracy on shorter phrases, the min and max character count was changed to 10 - 40, and similar results can be seen for major languages, but less known languages suffer significantly:

Language (102)	Symbol (alternates)	Count (837539)	Accuracy (10 - 40 chars)	Mislabels
Thai	th	3399	1
Malayalam	ml	525	1
Burmese	my	243	1
Tamil	ta	229	1
Telugu	te	220	1
Punjabi (Eastern)	pa	156	1
Amharic	am	154	1
Kannada	kn	126	1
Gujarati	gu	116	1
Sinhala	si	37	1
Tibetan	bo	29	1
Divehi, Dhivehi, Maldivian	dv	15	1
Japanese	ja	28060	0.9999643620812545	zh
Greek	el	24980	0.9999599679743795	en
Hebrew	he	26461	0.9999244170666264	en,yi
Korean	ko	6128	0.9996736292428199	tr,ja
Armenian	hy	1855	0.9994609164420485	de
Bengali	bn	4132	0.9992739593417231	bpy,as
Marathi	mr	25633	0.9989466703078064	hi,gom,pt,new
English	en	17094	0.9986544986544986	nl,it,hu,eo,es
Mandarin Chinese	zh	17801	0.9978652884669401	wuu,yue,ja,sr,pt
Turkish	tr	18879	0.9978282748026909	en,eo,az,es,it
Russian	ru	20855	0.9977942939343083	uk,bg,mk,sr,be
German	de	17223	0.9974452766649248	en,it,fr,es,sv
Uyghur	ug	6135	0.9973920130399349	ar,ba,tt,ca,hu
Vietnamese	vi	13130	0.9971058644325971	it,pms,eo,pt,fr
Esperanto	eo	21641	0.9966729818400258	it,es,tr,pt,pl
Georgian	ka	4550	0.996043956043956	xmf,en
Hindi	hi	11497	0.9958249978255197	mr,dty,new,bh,ne
Italian	it	20449	0.995598806787618	es,en,fr,eo,pt
Arabic	ar	25531	0.9955348399984333	arz,fa,en,mzn,ps
French	fr	16040	0.9953865336658354	en,it,ia,es,pt
Hungarian	hu	20843	0.9952502039053879	en,pt,it,nl,eo
Lao	lo	183	0.994535519125683	el
Polish	pl	21386	0.9940147760216964	en,it,eo,de,cs
Khmer	km	1252	0.9920127795527156	az,ru,sr,et
Spanish	es	20498	0.9895599570689824	pt,it,fr,ca,en
Finnish	fi	20731	0.9849500747672567	it,en,eo,et,nl
Portuguese	pt	18352	0.9833805579773321	es,it,gl,fr,en
Macedonian	mk	23602	0.9830099144140327	ru,bg,sr,uk
Ukrainian	uk	23251	0.982667412154316	ru,mk,bg,be,sr
Urdu	ur	1583	0.9797852179406191	pnb,fa,ug,en,ro
Dutch	nl	19349	0.9720915809602564	en,de,nds,af,fr
Lithuanian	lt	24184	0.9597667879589812	eo,fi,sr,pt,pl
Czech	cs	25189	0.951605859700663	sk,pl,hu,en,sl
Chuvash	cv	1332	0.9481981981981982	ru,uk,krc,ba,sr
Tatar	tt	8283	0.9471206084751902	ru,tr,az,kk,ky
Swedish	sv	24466	0.9464563067113545	da,no,en,de,eo
Icelandic	is	7745	0.9449967721110394	da,et,cs,no,de
Bulgarian	bg	19328	0.9352235099337748	mk,ru,uk,sr,tg
Sanskrit	sa	135	0.9259259259259259	hi,ne,mr
Kazakh	kk	2373	0.9258322798145807	uk,tt,tr,ru,ky
Romanian	ro	18367	0.9235041106332008	it,es,en,fr,pt
Tagalog	tl	11133	0.9193389023623462	ceb,en,it,id,es
Ossetian	os	205	0.9170731707317074	ru,hy,sr,kv,mrj
Indonesian	id	9707	0.9138765839085197	ms,en,it,eo,tr
Danish	da	22539	0.9081591907360576	no,sv,de,en,fr
Latin	la	24699	0.8979310903275436	it,fr,en,es,pt
Basque	eu	4570	0.8851203501094091	it,id,hu,nl,eo
Belarusian	be	9005	0.8785119378123265	ru,uk,bg,mk,pl
Cornish	kw	3757	0.8759648655842428	en,de,cy,es,br
Tajik	tg	48	0.875	ru,uk
Latvian	lv	2198	0.8735213830755232	lt,es,sr,en,fr
Breton	br	5468	0.8579005120702268	en,fr,pt,de,eu
Irish	ga	1977	0.840161861406171	en,pt,es,ca,gd
Bashkir	ba	128	0.8359375	tt,ru,sr,av,kk
Sindhi	sd	6	0.8333333333333334	ur
Serbian	sr	23128	0.8054738844690419	hr,mk,sh,ru,sl
Estonian	et	3077	0.8043548911277218	fi,en,hu,tr,it
Scottish Gaelic	gd	753	0.7822045152722443	en,ga,de,fr,pam
Welsh	cy	1167	0.7660668380462725	en,es,kw,la,it
Volapük	vo	3941	0.7609743719868054	id,en,eo,fi,de
Kyrgyz	ky	227	0.7533039647577092	ru,kk,tt,mn,bg
Catalan	ca	5313	0.7504234895539243	es,pt,it,fr,en
Assamese	as	2635	0.7127134724857686	bn,bpy,en,tl,bh
Yoruba	yo	31	0.7096774193548387	ga,pl,en,qu,ckb
Occitan	oc	4096	0.70751953125	es,fr,ca,pt,it
Interlingua	ia	14949	0.7073382834972239	it,es,fr,en,la
Afrikaans	af	3299	0.6808123673840558	nl,en,de,fr,nds
Norwegian Nynorsk	nn (no)	1287	0.6798756798756799	da,sv,de,es,hu
Maltese	mt	165	0.6727272727272727	hu,en,es,it,pl
Slovak	sk	13877	0.6105786553289616	cs,pl,sl,no,sr
Chechen	ce	25	0.6	bg,sr,mn,ba,uk
Interlingue	ie	6538	0.5183542367696543	es,it,en,fr,eo
Ido	io	6495	0.4857582755966128	eo,es,it,pt,tr
Slovenian	sl	908	0.46255506607929514	sr,hr,cs,pl,bs
Javanese	jv	548	0.45255474452554745	id,en,ko,ms,hu
Turkmen	tk	4585	0.45169029443838604	tr,en,uz,et,pl
Croatian	hr	4186	0.4362159579550884	sr,sh,bs,sl,pl
Galician	gl	3245	0.4200308166409861	pt,es,it,en,fr
Luxembourgish	lb	732	0.3975409836065574	de,fr,en,nds,nl
Frisian	fy	282	0.36879432624113473	nl,en,nds,de,fr
Walloon	wa	37	0.2972972972972973	fr,en,no,it,gn
Corsican	co	13	0.23076923076923078	it,min,ro,ilo,id
Sundanese	su	18	0.2222222222222222	id,es,en,it,lmo
Somali	so	61	0.14754098360655737	en,fi,et,cy,su
Limburgan, Limburger, Limburgish	li	34	0.14705882352941177	de,nl,en,no,is
Haitian Creole	ht	58	0.1206896551724138	en,fr,br,la,de
Manx	gv	30	0.06666666666666667	en,it,pt,fr,kw
Bosnian	bs	520	0.04423076923076923	sr,hr,sh,it,pl
Aragonese	an	73	0.0136986301369863	es,pt,it,en,fr
Romansh	rm	11	0	it,pt,fr,en,tl

Additional Insights

• During testing, the highest incorrect probability was often near 1, which means it's not possible to use a high possibility to suggest a correct assessment

• The lowest probability for a correct assessment varried widely. Although these were good predictors for some of the very accurate languages (99.9%), other languages were sometimes as low as a .09 probability. This means it's not possible to use a low probability as an accurate assessment of a false positive.

• To improve expectations of an incorrect result, you can use the difference in probability of result 1 and 2. It appears that the verage probability difference between 1 and 2 is somewhat of an indicator of a potentially incorrect prediction.

• Anything over 100 characters is strongly accurate, though there isn't enough sentences for test data to assure this for all the test languages. 55 out of 82 languages that had this data had a 99% or better accuracy, 63 had 90%+ accuracy, 72 had 75%+ accuracy. For 200+ characters, 42 of 47 languages had a perfect score, though most had less than 10 test cases.

• Spanish tends to give the most false positives based on sheer quantity of percentage of false positives.

• In attempting to add a second check with franc for a smaller difference in probabilities between language 1 and 2 (i.e. less than 0.2), only the worst performing languages showed significant benefit. There doesn't seem to be a trend for any other languages. You can see this data on the COMPARISONS.md.

Improving Accuracy

Most incorrect suggestions are due to non-text characters (i.e. punctuation) that should be filtered out to provide better results. Please submit an issue for incorrect suggestions so we can work on improving the accuracy.

Comparison NPM Libaries

Success benchmarking has been checked with other popular libraries (notably franc and languagedetect) and results are included in benchmark-testing/results/COMPARISONS.md

Sample Dockerfile

Note: You need to have python installed to make this work in alpine-node

FROM mhart/alpine-node:14

WORKDIR /usr/src/app

COPY package*.json ./

RUN apk add --no-cache --virtual .gyp \
  python \
  make \
  g++ \
  && npm ci --only=production \
  && apk del .gyp

COPY . ./

CMD [ "npm", "start" ]

TODO List

• Improve accuracy by replicating the test analysis from https://towardsdatascience.com/benchmarking-language-detection-for-nlp-8250ea8b67c and attempt to improve the formatText() function by strategically choosing punctuation / non-text characters.

This is an improved modification of https://www.npmjs.com/package/fasttext-lid

Created with <3 for https://smodin.io