Rob Speer 963e0ff785 refactor the tokenizer, add include_punctuation option ... Former-commit-id: e8e6e0a231		2015-09-15 13:26:09 -04:00
..
test_chinese.py	add external_wordlist option to tokenize	2015-09-10 18:09:41 -04:00
test_japanese.py	Lower the frequency of phrases with inferred token boundaries	2015-09-10 14:16:22 -04:00
test.py	refactor the tokenizer, add include_punctuation option	2015-09-15 13:26:09 -04:00