iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-24 09:51:38 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

start a new multilingual wordlist called 'stems'

Browse Source 
So far, this wordlist is only in Dutch.


Former-commit-id: af5f65b328
This commit is contained in:
Rob Speer 2015-03-31 15:59:30 -04:00
parent 58da7797da
commit d3c41fd8d8
5 changed files with 44611 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
wordfreq/build.py
View File

@ -193,11 +193,10 @@ def load_all_data(source_dir=None, filename=None, do_it_anyway=False):
logger.info("Loading combined multilingual corpus:")
multi_wordlist = read_multilingual_csv(wordlist_path('luminoso', 'multilingual.csv'))
for lang in multi_wordlist:
if lang != 'nl':
logger.info("\tLanguage: %s" % lang)
save_wordlist_to_db(conn, 'multi', lang, multi_wordlist[lang])
logger.info("\tLanguage: %s" % lang)
save_wordlist_to_db(conn, 'multi', lang, multi_wordlist[lang])
# Load Dutch from a separate source. We may end up with more languages like this.
read_wordlist_into_db(conn, wordlist_path('luminoso', 'nl-combined-201503.csv'), 'multi', '*')
read_wordlist_into_db(conn, wordlist_path('luminoso', 'nl-combined-201503.csv'), 'stems', '*')
logger.info("Done loading.")

2
wordfreq_data/luminoso/nl-combined-201503.csv.REMOVED.git-id
View File

@ -1 +1 @@
acde5e04c02b070c9f5c90b1da73ce6cb91bf937
9b29de132c82bd7287c08c2937e3c4821525e356

21282
wordfreq_data/luminoso/twitter-stems-2014-nl.csv Normal file
View File

File diff suppressed because it is too large Load Diff

23324
wordfreq_data/luminoso/twitter-surfaces-2014-nl.csv Normal file
View File

File diff suppressed because it is too large Load Diff

1
wordfreq_data/wikipedia/stems-nl.csv.REMOVED.git-id Normal file
View File

@ -0,0 +1 @@
b9d52d81bbe078a7de17519ed3494eb4771f0f69