iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-26 02:28:50 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
340 Commits 8 Branches 124 Tags 707 MiB
c11e3b7a9d
Commit Graph

120 Commits

Author SHA1 Message Date
Rob Speer
531db64288 Note on next languages to support 2015-09-04 01:50:15 -04:00
Rob Speer
d94428d454 support Turkish and more Greek; document more 2015-09-04 00:57:04 -04:00
Rob Speer
45d871a815 Merge branch 'add-subtlex' into greek-and-turkish 2015-09-03 23:26:14 -04:00
Rob Speer
40d82541ba refer to merge_freqs command correctly 2015-09-03 23:25:46 -04:00
Rob Speer
a3daba81eb expand Greek and enable Turkish in config 2015-09-03 23:23:31 -04:00
Rob Speer
2d58ba94f2 Add SUBTLEX as a source of English and Chinese data 
Meanwhile, fix up the dependency graph thingy. It's actually kind of
legible now.
 2015-09-03 18:13:13 -04:00
Rob Speer
5def3a7897 update the build diagram and its script 2015-08-28 17:47:04 -04:00
Rob Speer
c4a2594217 fix URL expression 2015-08-26 15:00:46 -04:00
Rob Speer
a893823d6e un-flake wordfreq_builder.tokenizers, and edit docstrings 2015-08-26 13:03:23 -04:00
Rob Speer
5a1fc00aaa Strip apostrophes from edges of tokens 
The issue here is that if you had French text with an apostrophe,
such as "d'un", it would split it into "d'" and "un", but if "d'"
were re-tokenized it would come out as "d". Stripping apostrophes
makes the process more idempotent.
 2015-08-25 12:41:48 -04:00
Rob Speer
de73888a76 use better regexes in wordfreq_builder tokenizer 2015-08-24 19:05:46 -04:00
Rob Speer
140ca6c050 remove Hangul fillers that confuse cld2 2015-08-24 17:11:18 -04:00
Andrew Lin
6d40912ef9 Stylistic cleanups to word_counts.py. 2015-07-31 19:26:18 -04:00
Andrew Lin
53621c34df Remove redundant reference to wikipedia in builder README. 2015-07-31 19:12:59 -04:00
Rob Speer
e9f9c94e36 Don't use the file-reading cutoff when writing centibels 2015-07-28 18:45:26 -04:00
Rob Speer
c5708b24e4 put back the freqs_to_cBpack cutoff; prepare for 1.0 2015-07-28 18:01:12 -04:00
Rob Speer
32102ba3c2 Merge pull request #19 from LuminosoInsight/code-review-fixes-2015-07-17 
Code review fixes 2015 07 17
 2015-07-22 15:09:00 -04:00
Joshua Chin
93cd902899 updated read_freqs docs 2015-07-22 10:06:16 -04:00
Joshua Chin
4fe9d110e1 fixed style 2015-07-22 10:05:11 -04:00
Joshua Chin
6453d864c4 reordered command line args 2015-07-22 10:04:14 -04:00
Joshua Chin
8081145922 bugfix 2015-07-21 10:12:56 -04:00
Joshua Chin
c5f82ecac1 fixed rules.ninja 2015-07-20 17:20:29 -04:00
Joshua Chin
643571c69c fixed build bug 2015-07-20 16:51:25 -04:00
Joshua Chin
173278fdd3 ensure removal of tatweels (hopefully) 2015-07-20 16:48:36 -04:00
Joshua Chin
298d3c1d24 unhoisted if statement 2015-07-20 11:10:41 -04:00
Joshua Chin
accb7e398c ninja.py is now pep8 compliant 2015-07-20 11:06:58 -04:00
Joshua Chin
221acf7921 fixed build 2015-07-17 17:44:01 -04:00
Rob Speer
2d1020daac mention the Wikipedia data, and credit Hermit Dave 2015-07-17 17:09:36 -04:00
Joshua Chin
f31f9a1bcd fixed tokenize_twitter 2015-07-17 16:37:47 -04:00
Joshua Chin
a44927e98e added cld2 tokenizer comments 2015-07-17 16:03:33 -04:00
Joshua Chin
11a1c51321 fix arabic tokens 2015-07-17 15:52:12 -04:00
Joshua Chin
c75c735d8d fixed syntax 2015-07-17 15:43:24 -04:00
Joshua Chin
303bd88ba2 renamed tokenize file to tokenize twitter 2015-07-17 15:27:26 -04:00
Joshua Chin
d6519cf736 created last_tab flag 2015-07-17 15:19:09 -04:00
Joshua Chin
620becb7e8 removed uncessary if statement 2015-07-17 15:14:06 -04:00
Joshua Chin
d988b1b42e generated freq dict in place 2015-07-17 15:13:25 -04:00
Joshua Chin
e37c689031 corrected docstring 2015-07-17 15:12:23 -04:00
Joshua Chin
002351bace removed unnecessary strip 2015-07-17 15:11:28 -04:00
Joshua Chin
7fc23666a9 moved last_tab to tokenize_twitter 2015-07-17 15:10:17 -04:00
Joshua Chin
528285a982 removed unused function 2015-07-17 15:03:14 -04:00
Joshua Chin
59d3c72758 fixed spacing 2015-07-17 15:02:34 -04:00
Joshua Chin
10028be212 removed unnecessary format 2015-07-17 15:01:25 -04:00
Joshua Chin
3b368b66dd cleaned up BAD_CHAR_RANGE 2015-07-17 15:00:59 -04:00
Joshua Chin
c2d1cdcb31 moved test tokenizers 2015-07-17 14:58:58 -04:00
Joshua Chin
5d26c9f57f added docstring and moved to scripts 2015-07-17 14:56:18 -04:00
Joshua Chin
bdc791af8f style changes 2015-07-17 14:54:32 -04:00
Joshua Chin
4d5ec57144 removed bad comment 2015-07-17 14:54:09 -04:00
Joshua Chin
39f01b0485 removed unused scripts 2015-07-17 14:53:18 -04:00
Joshua Chin
98a7a8093b removed mkdir -p for many cases 2015-07-17 14:45:22 -04:00
Joshua Chin
449a656edd removed TOKENIZE_TWITTER 2015-07-17 14:43:14 -04:00