iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 09:21:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Mention that multi-digit numbers are combined together

Browse Source
This commit is contained in:
Robyn Speer 2017-01-05 19:24:28 -05:00
parent 48a5967e9a
commit 86f22e8523
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
CHANGELOG.md
View File

@ -14,6 +14,9 @@
- Add automatic transliteration of Serbian text
- Adjust tokenization of apostrophes next to vowel sounds: the French word
"l'heure" is now tokenized similarly to "l'arc"
- Numbers longer than a single digit are smashed into the same word frequency,
to remove meaningless differences and increase compatibility with word2vec.
(Internally, their digits are replaced by zeroes.)
- Another new frequency-merging strategy (drop the highest and lowest,
average the rest)