iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-24 01:41:39 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
420 Commits 8 Branches 124 Tags 707 MiB
1942bc690f
Commit Graph

3 Commits

Author SHA1 Message Date
Robyn Speer
af29fc4f88 fix URL expression 
Former-commit-id: c4a2594217
 2015-08-26 15:00:46 -04:00
Robyn Speer
b22a4b0f02 Strip apostrophes from edges of tokens 
The issue here is that if you had French text with an apostrophe,
such as "d'un", it would split it into "d'" and "un", but if "d'"
were re-tokenized it would come out as "d". Stripping apostrophes
makes the process more idempotent.


Former-commit-id: 5a1fc00aaa
 2015-08-25 12:41:48 -04:00
Joshua Chin
78e9cf5d8f moved test tokenizers 
Former-commit-id: c2d1cdcb31
 2015-07-17 14:58:58 -04:00