utoken is a universal tokenizer (multilingual word segmenter) that divides text into words, punctuation and special tokens such as numbers, URLs, XML tags, email-addresses and hashtags. It comes with a companion detokenizer.
utoken is a multilingual tokenizer that divides text into words, punctuation and special tokens such as numbers, URLs, XML tags, email-addresses and hashtags. The tokenizer comes with a companion detokenizer. Initial public release of beta version 0.1.0 on Oct. 1, 2021.
Capt. O'Connor's car can't've cost $100,000.
Capt. O'Connor 's car can n't 've cost $ 100,000 .
The ouput below is in the more human-friendly annotation format. Default format is the more computer-friendly JSON.
::line 1 ::s Capt. O'Connor's car can't've cost$100,000.
::span 0-5 ::type ABBREV ::sem-class military-rank ::surf Capt.
::span 6-14 ::type LEXICAL ::sem-class person-last-name ::surf O'Connor
::span 14-16 ::type DECONTRACTION ::surf 's
::span 17-20 ::type WORD-B ::surf car
::span 21-23 ::type DECONTRACTION ::surf can
::span 23-26 ::type DECONTRACTION ::surf n't
::span 26-29 ::type DECONTRACTION-R ::surf 've
::span 30-34 ::type WORD-B ::surf cost
::span 34-35 ::type PUNCT ::sem-class currency-unit ::surf $
::span 35-42 ::type NUMBER ::surf 100,000
::span 42-43 ::type PUNCT-E ::surf .
python -m utoken.utokenize [-h] [-i INPUT-FILENAME] [-o OUTPUT-FILENAME] [-a ANNOTATION-FILENAME]
[--annotation_format ANNOTATION_FORMAT] [-p PROFILE-FILENAME]
[--profile_scope PROFILE_SCOPE] [-d DATA_DIRECTORY] [--lc LANGUAGE-CODE]
[-f] [-v] [-c] [--simple] [--version]
or simply
utokenize [-h] [-i INPUT-FILENAME] [-o OUTPUT-FILENAME] [-a ANNOTATION-FILENAME]
[--annotation_format ANNOTATION_FORMAT] [-p PROFILE-FILENAME]
[--profile_scope PROFILE_SCOPE] [-d DATA_DIRECTORY] [--lc LANGUAGE-CODE]
[-f] [-v] [-c] [--simple] [--version]
optional arguments:
-h, --help show this help message and exit
-i INPUT-FILENAME, --input INPUT-FILENAME
(default: STDIN)
-o OUTPUT-FILENAME, --output OUTPUT-FILENAME
(default: STDOUT)
-a ANNOTATION-FILENAME, --annotation_file ANNOTATION-FILENAME
(optional output)
--annotation_format ANNOTATION_FORMAT
(default: 'json'; alternative: 'double-colon')
-p PROFILE-FILENAME, --profile PROFILE-FILENAME
(optional output for performance analysis)
--profile_scope PROFILE_SCOPE
(optional scope for performance analysis)
-d DATA_DIRECTORY, --data_directory DATA_DIRECTORY
(default: standard data directory)
--lc LANGUAGE-CODE ISO 639-3, e.g. 'fas' for Persian
-f, --first_token_is_line_id
First token is line ID (and will be exempt from any tokenization)
-v, --verbose write change log etc. to STDERR
-c, --chart build annotation chart, even without annotation output
--simple prevent MT-style output (e.g. @-@). Note: can degrade any detokenization
--version show program's version number and exit
Note: Please make sure that your $PYTHONPATH includes the directory in which this README file resides.
Example:
echo "Wait,dont tell me." | utokenize --lc eng
Output:
Wait , do n't tell me .
python -m utoken.detokenize [-h] [-i INPUT-FILENAME] [-o OUTPUT-FILENAME] [-d DATA_DIRECTORY]
[--lc LANGUAGE-CODE] [-f] [-v] [--version]
or simply
detokenize [-h] [-i INPUT-FILENAME] [-o OUTPUT-FILENAME] [-d DATA_DIRECTORY]
[--lc LANGUAGE-CODE] [-f] [-v] [--version]
optional arguments:
-h, --help show this help message and exit
-i INPUT-FILENAME, --input INPUT-FILENAME
(default: STDIN)
-o OUTPUT-FILENAME, --output OUTPUT-FILENAME
(default: STDOUT)
-d DATA_DIRECTORY, --data_directory DATA_DIRECTORY
(default: standard data directory)
--lc LANGUAGE-CODE ISO 639-3, e.g. 'fas' for Persian
-f, --first_token_is_line_id
First token is line ID (and will be exempt from any tokenization)
-v, --verbose write change log etc. to STDERR
--version show program's version number and exit
Note: Please make sure that your $PYTHONPATH includes the directory in which this README file resides.
Example:
echo "Wait , do n't tell me ." | detokenize --lc eng
Output:
Wait, don't tell me.
from utoken import utokenize
tok = utokenize.Tokenizer(lang_code='eng') # Initialize tokenizer, load resources
print(tok.utokenize_string("Dont worry!"))
print(tok.utokenize_string("Sold,for $9,999.99 on ebay.com."))
Output:
Do n't worry !
Sold , for $ 9,999.99 on ebay.com .
Note: Please make sure that your $PYTHONPATH includes the directory in which this README file resides.
from utoken import detokenize
detok = detokenize.Detokenizer(lang_code='eng') # Initialize detokenizer, load resources
print(detok.detokenize_string("Do n't worry !"))
print(detok.detokenize_string("Sold , for $ 9,999.99 on ebay.com ."))
Output:
Don't worry!
Sold, for $9,999.99 on ebay.com.
Note: Please make sure that your $PYTHONPATH includes the directory in which this README file resides.
pip install utoken
or
git clone https://github.com/uhermjakob/utoken.git
import regexJohn's → John 's; we've → we 've; can't → can n't; won't → will n't5,000km² → 5,000 km²350th → 350 thpeace-loving → peace @-@ lovingJ.S.Bach → J. S. Bach<a href="http://www.hollywoodbowl.com">https://www.youtube.com/watch?v=AaZ_RSt0KP8а.almukhanov@energo.gov.kzОперплан_каз2015.doc-12,345,678.90 १,२३,४५,६७८.९०Mr. e.g. w/oT-shirt father-in-law so-called Port-au-PrinceXi’an 's-Gravenhage#global_warming #2 @GermanBeer???⚽👍🎉میخواهمFor many application such as machine translation, tokenization is important, but should be reversed when producing the final output.
In some cases, this is relatively straight forward, so . and , typically attach to the word on the left and ( attaches to the word on the right.
In other cases, it can generally be very hard to decide how to detokenize, so we add a special tag such as @ during tokenization in order to guide later dekonization.
A @ on one or both sides of punctuation indicates that in the original text, the punctuation and neighboring word were together.
To look at it in another way, the tokenizer basically upgrades the non-directional " to an open "@ or close @" delimiter.
Example: ("Hello,world!") Tokenized: ( "@ Hello , world ! @" ) Detokenized: ("Hello, world!")
If later detokenization is not import and you want to suppress any markup with @, call utokenizer.py with the option --simple
Example: ("Hello,world!") Tokenized (simple): ( " Hello , world ! " ) Detokenized: (" Hello, world! ")
In some applications, the text to be tokenized is preceded by a sentence ID at the beginning of each line and tokenization should not be applied to those sentence IDs.
Option --first_token_is_line_id, or -f for short, suppresses tokenization of those sentence IDs.
GEN:1:1 In the beginning, God created the heavens and the earth.utokenize.pl tokenization: GEN @:@ 1 @:@ 1 In the beginning , God created the heavens and the earth .utokenize.pl -f tokenization: GEN:1:1 In the beginning , God created the heavens and the earth .Tokenization is more then just splitting a sentence along spaces, as a lot of punctuation such as commas and periods are attached to adjacent words.
But we can't just blindly split off commas and periods, as this would break numbers such as 12,345.60, abbreviations such as Mr. or URLs such as www.usc.edu.
<a href="http://www.hollywoodbowl.com">https://www.youtube.com/watch?v=AaZ_RSt0KP8а.almukhanov@energo.gov.kzОперплан_каз2015.doc-12,345,678.90 १,२३,४५,६७८.९०#global_warming #2 @GermanBeerMr. e.g. w/osu'aal (question). Apostrophes can appear in foreign names (e.g. Xi'an and 's-Gravenhage). In some languages, an apostrophe is used for contractions, such as John's and we'll in English. Additionally, an apostrophe can be used as a quote around a word or phrase such as 'Good job!'. All these cases have to be treated differently.peace-loving (which should be split). But they also occur inside lexical phrases such as T-shirt that should not be split.zero width non-joiner.utokenize includes a number of data files to supports its operation:
tok-resource.txt includes language-independent tokenization resource entries, especially for punctuation, abbreviations (e.g. km²) and names (especially those with hyphens, spaces and other non-alpha characters)tok-resource-eng-global.txt contains tokenization resource entries for English that are also loaded for other languages. This is helpful as foreign texts often code-switch to English.tok-resource-eng.txt contains tokenization resource entries for English that are not shared, including those that would not work in other languages. For example, in English, dont in a non-standard version of don't and is tokenized into do n't, but in French, dont (of which) is a regular word that should be left alone.detok-resource.txt includes resources for detokenization. The file is also used by the tokenizer to mark up certain punctuation with attachment tags such as @-@.tok-resource-xxx.txt files for other languages, some larger than others. Some languages such as Farsi just don't use contractions and abbreviations with periods that much, so there are few entries. Others files might benefit from additional contributions.top-level-domain-codes.txt contains a list of suffixes such as .com, .org, .uk, .tv to support tokenization of URLs and email address.Exmaples of resource entries:
::punct-split ! ::side end ::group True ::comment multiple !!! remain grouped as a single token
::contraction can't ::target can n't ::lcode eng
::repair wo n't ::target will n't ::lcode eng ::problem previous tokenizer
::abbrev No. ::exp number ::lcode eng ::sem-class corpus-component ::case-sensitive True ::right-context \s*\d
::lexical T-shirt ::lcode eng ::plural +s
::misspelling accomodate ::target accommodate ::lcode eng ::suffix-variations e/ed;es;ing;ion;ions
::markup-attach - ::group True ::comment hyphen-minus ::example the hyphen in _peace-loving_ will be marked up as ```@-@```
::auto-attach th ::side left ::left-context \d ::lcode eng ::example 20th
210,000 characters per second (real time) on a 39k sentence English AMR corpus on a 2021 MacBook Pro using a single CPU. Parallelization is trivial as sentences are tokenized independent of each other.
utoken has been tested on 71 corpora in 60 languages and 18 scripts (as of Oct. 18, 2021). Tests include
Plans include
FAQs
utoken is a universal tokenizer (multilingual word segmenter) that divides text into words, punctuation and special tokens such as numbers, URLs, XML tags, email-addresses and hashtags. It comes with a companion detokenizer.
We found that utoken demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
An advanced npm supply chain attack is leveraging Ethereum smart contracts for decentralized, persistent malware control, evading traditional defenses.
Security News
Research
Attackers are impersonating Sindre Sorhus on npm with a fake 'chalk-node' package containing a malicious backdoor to compromise developers' projects.
Security News
The npm package for the LottieFiles Player web component was hit with a supply chain attack after a software engineer's npmjs credentials were compromised.