Robyn Speer abd0820a32 Handle smashing numbers only at the end of tokenize(). ... This does make the code a lot clearer.		2017-01-11 19:04:19 -05:00
..
data	update data from Exquisite Corpus in English and Swedish	2017-01-05 19:17:51 -05:00
__init__.py	Don't smash numbers in *all* tokenization, just when looking up freqs	2017-01-06 19:18:52 -05:00
chinese.py	Merge branch 'master' into chinese-external-wordlist	2015-09-28 14:34:59 -04:00
mecab.py	Allow MeCab to work in Japanese or Korean without the other	2016-08-19 11:41:35 -04:00
tokens.py	Handle smashing numbers only at the end of tokenize().	2017-01-11 19:04:19 -05:00
transliterate.py	transliterate: organize the 'borrowed letters' better	2017-01-05 13:23:20 -05:00