iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 17:31:41 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
498 Commits 8 Branches 124 Tags 707 MiB
71a0ad6abb
Commit Graph

1 Commits

Author SHA1 Message Date
Robyn Speer
0a2bfb2710 Tokenization in Korean, plus abjad languages (#38) 
* Remove marks from more languages

* Add Korean tokenization, and include MeCab files in data

* add a Hebrew tokenization test

* fix terminology in docstrings about abjad scripts

* combine Japanese and Korean tokenization into the same function


Former-commit-id: fec6eddcc3
 2016-07-15 15:10:25 -04:00