iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-25 18:18:53 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
255 Commits 8 Branches 124 Tags 707 MiB
b0a9a2980f
Commit Graph

255 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rob Speer
05cf94d1fd Work on making Japanese tokenization use MeCab consistently 2015-05-27 18:10:25 -04:00
Rob Speer
cbe3513e08 Tokenize Japanese consistently with MeCab 2015-05-27 17:44:58 -04:00
Rob Speer
536c15fbdb give mecab a larger buffer 2015-05-26 19:34:46 -04:00
Rob Speer
5de81c7111 fix build rules for Japanese Wikipedia 2015-05-26 18:08:57 -04:00
Rob Speer
3d5b3d47e8 fix version in config.py 2015-05-26 18:08:46 -04:00
Rob Speer
ffd352f148 correct a Leeds bug; add some comments to rules.ninja 2015-05-26 18:08:04 -04:00
Rob Speer
0e5156e162 Merge branch 'master' into newbuild 
Conflicts:
	setup.py
	wordfreq/build.py
	wordfreq/config.py
 2015-05-21 20:41:47 -04:00
Rob Speer
84e5edcea1 rebuild data 2015-05-21 20:36:15 -04:00
Rob Speer
410912d8f0 remove old tests 2015-05-21 20:36:09 -04:00
Rob Speer
b42594fa5f allow more language matches; reorder some parameters 2015-05-21 20:35:02 -04:00
Rob Speer
df863a5169 tests for new wordfreq with full coverage 2015-05-21 20:34:17 -04:00
Rob Speer
dd41e61c57 update README, another setup fix 2015-05-13 04:09:34 -04:00
Rob Speer
f13cca4d81 update dependencies 2015-05-12 12:30:01 -04:00
Rob Speer
bb18f741e2 restore missing line in setup.py 2015-05-12 12:24:18 -04:00
Rob Speer
35aec061de add new data files from wordfreq_builder 2015-05-11 18:45:47 -04:00
Rob Speer
50ff85ce19 add Google Books data for English 2015-05-11 18:44:28 -04:00
Rob Speer
c707b32345 move some functions to the wordfreq package 2015-05-11 17:02:52 -04:00
Rob Speer
9b63e54471 WIP: burn stuff down 2015-05-08 15:28:52 -04:00
Rob Speer
d0d777ed91 use a more general-purpose tokenizer, not 'retokenize' 2015-05-08 12:40:14 -04:00
Rob Speer
35128a94ca build.ninja knows about its own dependencies 2015-05-08 12:40:06 -04:00
Rob Speer
d6cc90792f Makefile should only be needed for bootstrapping Ninja 2015-05-08 12:39:31 -04:00
Rob Speer
b541fe68e1 Merge branch 'ninja-build' 
Conflicts:
	wordfreq_builder/cmd_count_twitter.py
	wordfreq_builder/cmd_count_wikipedia.py
 2015-05-08 00:01:01 -04:00
Rob Speer
2f14417bcf limit final builds to languages with >= 2 sources 2015-05-07 23:59:04 -04:00
Rob Speer
1b7a2b9d0b fix dependency 2015-05-07 23:55:57 -04:00
Rob Speer
abb0e059c8 a reasonably complete build process 2015-05-07 19:38:33 -04:00
Rob Speer
02d8b32119 process leeds and opensubtitles 2015-05-07 17:07:33 -04:00
Rob Speer
7e238cf547 abstract how we define build rules a bit 2015-05-07 16:59:28 -04:00
Rob Speer
d2f9c60776 WIP on more build steps 2015-05-07 16:49:53 -04:00
Lance Nathan
e8a1548d93 Tweak to previous variable name fix 2015-05-06 17:57:10 -04:00
Lance Nathan
4632ffb177 Merge pull request #6 from LuminosoInsight/ftfy4 
Clean data with ftfy v4
 2015-05-06 17:32:45 -04:00
Lance Nathan
5f05b52fe5 Merge pull request #5 from LuminosoInsight/dutch-201504 
Better Dutch surface-form data
 2015-05-06 17:15:21 -04:00
Rob Speer
506073030a fix reused variable name 2015-05-06 17:06:37 -04:00
Rob Speer
16928ed182 add rules to count wikipedia tokens 2015-05-05 15:21:24 -04:00
Rob Speer
bd579e2319 fix the 'count' ninja rule 2015-05-05 14:06:13 -04:00
Rob Speer
5787b6bb73 add and adjust some build steps 
- more build steps for Wikipedia
- rename 'tokenize_twitter' to 'pretokenize_twitter' to indicate that
  the results are preliminary
 2015-05-05 13:59:21 -04:00
Rob Speer
2f3bb955d1 set version number to 0.8 2015-05-05 12:05:00 -04:00
Rob Speer
24a7c73e6d Merge branch 'dutch-201504' into ftfy4 
Conflicts:
	setup.py
 2015-05-05 12:04:44 -04:00
Rob Speer
70b2c678ea require ftfy 4 2015-05-05 12:04:13 -04:00
Rob Speer
61b9440e3d add wiki-parsing process 2015-05-04 13:25:01 -04:00
Rob Speer
34400de35a not using wordfreq.cfg anymore 2015-04-30 16:25:42 -04:00
Rob Speer
5437bb4e85 WIP on new build system 2015-04-30 16:24:28 -04:00
Rob Speer
2a1b16b55c use script codes for Chinese 2015-04-30 13:02:58 -04:00
Rob Speer
873ace87db v0.7: make a proper Dutch 'surfaces' list 2015-04-30 13:01:24 -04:00
Rob Speer
4dae2f8caf define some ninja rules 2015-04-29 17:13:58 -04:00
Rob Speer
14e445a937 WIP on Ninja build automation 2015-04-29 15:59:06 -04:00
Rob Speer
815d393b74 move commands into cli/ directory 2015-04-29 15:22:04 -04:00
Rob Speer
5d14d24738 always use surface forms 2015-04-29 15:17:00 -04:00
Rob Speer
6cf46ee5aa Merge branch 'master' into dutch-201503 
Conflicts:
	wordfreq/build.py
 2015-04-29 14:36:24 -04:00
Rob Speer
70c9e99ee4 handle multi-word stems correctly 2015-04-29 13:45:53 -04:00
Rob Speer
af5f65b328 start a new multilingual wordlist called 'stems' 
So far, this wordlist is only in Dutch.
 2015-03-31 15:59:30 -04:00