iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-27 19:08:51 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
d29e8bfddf
wordfreq/wordfreq_data/luminoso
History
Robyn Speer d29e8bfddf start a new multilingual wordlist called 'stems' 
So far, this wordlist is only in Dutch.


Former-commit-id: af5f65b328
2015-03-31 15:59:30 -04:00
..
multilingual.csv.REMOVED.git-id try to match the wordlist metanl actually uses 2013-10-31 15:13:22 -04:00
nl-combined-201503.csv.REMOVED.git-id start a new multilingual wordlist called 'stems' 2015-03-31 15:59:30 -04:00
README.txt Add wordfreq_data files. 2013-10-31 13:39:02 -04:00
twitter-52M.csv.REMOVED.git-id Add wordfreq_data files. 2013-10-31 13:39:02 -04:00
twitter-stems-2014-nl.csv start a new multilingual wordlist called 'stems' 2015-03-31 15:59:30 -04:00
twitter-stems-2014.csv.REMOVED.git-id add twitter-stems-2014 wordlist data 2015-02-11 13:29:32 -05:00
twitter-surfaces-2014-nl.csv start a new multilingual wordlist called 'stems' 2015-03-31 15:59:30 -04:00
twitter-surfaces-2014.csv.REMOVED.git-id new Dutch data, bump version to 0.6 2015-03-03 15:54:45 -05:00

README.txt 

This directory contains two wordlists we've put together at Luminoso for our
own purposes. You might find them useful as well.

* `twitter-52M` collects the unigram word frequencies from 52 million tweets.
  The words are not distinguished by language.

* `multi` combines various sources of data in different languages, including:

  * Google Books, for English
  * A smaller corpus of tweets that supposedly come from English speakers
    (there's still a lot of non-English text in there)
  * the Leeds corpora for various languages (see `../leeds/README.txt`)

We would like to release the tools that built `twitter-52M` as soon as they are
less sloppy. `multi` is a dataset that is mainly relevant because it's the data
we happen to already be using, but you might find it useful as well.