iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2025-01-15 13:46:01 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
379 Commits 8 Branches 125 Tags 707 MiB
7a3ea2bf79
Commit Graph

2 Commits

Author SHA1 Message Date
Rob Speer
5a1fc00aaa Strip apostrophes from edges of tokens 
The issue here is that if you had French text with an apostrophe,
such as "d'un", it would split it into "d'" and "un", but if "d'"
were re-tokenized it would come out as "d". Stripping apostrophes
makes the process more idempotent.
 2015-08-25 12:41:48 -04:00
Joshua Chin
c2d1cdcb31 moved test tokenizers 2015-07-17 14:58:58 -04:00