Robyn Speer 573ecc53d0 Don't smash numbers in *all* tokenization, just when looking up freqs ... I forgot momentarily that the output of the tokenizer is used by other code.		2017-01-06 19:18:52 -05:00
..
test_chinese.py	import new wordlists from Exquisite Corpus	2017-01-05 17:59:26 -05:00
test_french_and_related.py	add a test for "aujourd'hui"	2016-12-06 17:39:40 -05:00
test_japanese.py	Revert a small syntax change introduced by a circular series of changes.	2015-09-24 13:24:11 -04:00
test_korean.py	Tokenization in Korean, plus abjad languages (#38)	2016-07-15 15:10:25 -04:00
test_serbian.py	transliterate: Handle unexpected Russian invasions	2017-01-04 18:51:00 -05:00
test.py	Don't smash numbers in *all* tokenization, just when looking up freqs	2017-01-06 19:18:52 -05:00