iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 09:21:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

mention tokenization change in changelog

Browse Source
This commit is contained in:
Robyn Speer 2017-01-05 19:19:31 -05:00
parent 39e459ac71
commit 48a5967e9a
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CHANGELOG.md
View File

@ -12,6 +12,8 @@
- Drop the Common Crawl; we have enough good sources now that we don't have
to deal with all that spam
- Add automatic transliteration of Serbian text
- Adjust tokenization of apostrophes next to vowel sounds: the French word
"l'heure" is now tokenized similarly to "l'arc"
- Another new frequency-merging strategy (drop the highest and lowest,
average the rest)