wordfreq/CHANGELOG.md at 1363f9d2e03429776475c78731c45a67ef46548f

iskm/wordfreq

mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 09:21:37 +00:00

Robyn Speer 93306e55a0 Update README with new examples and URL

2017-01-09 15:13:19 -05:00

Version 1.6.0 (2017-01-05)

Support Czech, Persian, Ukrainian, and Croatian/Bosnian/Serbian
Add large lists in Chinese, Finnish, Japanese, and Polish
Data is now collected and built using Exquisite Corpus (https://github.com/LuminosoInsight/exquisite-corpus)
Add word frequencies from OPUS OpenSubtitles 2016
Add word frequencies from the MOKK Hungarian Webcorpus
Expand Google Books Ngrams data to cover 8 languages
Expand language detection on Reddit to cover 13 languages with large enough Reddit communities
Drop the Common Crawl; we have enough good sources now that we don't have to deal with all that spam
Add automatic transliteration of Serbian text
Adjust tokenization of apostrophes next to vowel sounds: the French word "l'heure" is now tokenized similarly to "l'arc"
Numbers longer than a single digit are smashed into the same word frequency, to remove meaningless differences and increase compatibility with word2vec. (Internally, their digits are replaced by zeroes.)
Another new frequency-merging strategy (drop the highest and lowest, average the rest)

Bug fix: Made it possible to load the Japanese or Korean dictionary when the other one is not available

Include word frequencies learned from the Common Crawl
Support Bulgarian, Catalan, Danish, Finnish, Hebrew, Hindi, Hungarian, Norwegian Bokmål, and Romanian
Improve Korean with MeCab tokenization
New frequency-merging strategy (weighted median)
Include Wikipedia as a Chinese source (mostly Traditional)
Include Reddit as a Spanish source
Remove Greek Twitter because its data is poorly language-detected
Add large lists in Arabic, Dutch, Italian
Remove marks from more languages
Deal with commas and cedillas in Turkish and Romanian
Fix tokenization of Southeast and South Asian scripts
Clean up Git history by removing unused large files

Add SUBTLEX data
Better support for Chinese, using Jieba for tokenization, and mapping Traditional Chinese characters to Simplified
Improve Greek
Add Polish, Swedish, and Turkish
Tokenizer can optionally preserve punctuation
Detect when sources stripped "'t" off of English words, and repair their frequencies

Use the 'regex' package to implement Unicode tokenization that's mostly consistent across languages
Use NFKC normalization in Japanese and Arabic

Create compact word frequency lists in English, Arabic, German, Spanish, French, Indonesian, Japanese, Malay, Dutch, Portuguese, and Russian
Marginal support for Greek, Korean, Chinese
Fresh start, dropping compatibility with wordfreq 0.x and its unreasonably large downloads