iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-26 10:28:52 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
262 Commits 8 Branches 124 Tags 707 MiB
c2d1cdcb31
Commit Graph

262 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Joshua Chin
c2d1cdcb31 moved test tokenizers 2015-07-17 14:58:58 -04:00
Joshua Chin
5d26c9f57f added docstring and moved to scripts 2015-07-17 14:56:18 -04:00
Joshua Chin
bdc791af8f style changes 2015-07-17 14:54:32 -04:00
Joshua Chin
4d5ec57144 removed bad comment 2015-07-17 14:54:09 -04:00
Joshua Chin
39f01b0485 removed unused scripts 2015-07-17 14:53:18 -04:00
Joshua Chin
afaed8757b updated word_frequency docstring 2015-07-17 14:52:06 -04:00
Joshua Chin
131b916c57 updated comments 2015-07-17 14:50:12 -04:00
Joshua Chin
b0a9a2980f no use for use 2015-07-17 14:46:40 -04:00
Joshua Chin
98a7a8093b removed mkdir -p for many cases 2015-07-17 14:45:22 -04:00
Joshua Chin
449a656edd removed TOKENIZE_TWITTER 2015-07-17 14:43:14 -04:00
Joshua Chin
00e18b7d4b removed TOKENIZE_TWITTER option 2015-07-17 14:40:49 -04:00
Joshua Chin
772c0cddd1 more README fixes 2015-07-17 14:40:33 -04:00
Joshua Chin
0a085132f4 fixed README 2015-07-17 14:35:43 -04:00
Rob Speer
8633e8c2a9 update the wordfreq_builder README 2015-07-13 11:58:48 -04:00
Joshua Chin
668f0fa67f Merge ../wordfreq_builder 2015-07-13 11:03:18 -04:00
Joshua Chin
2c573b5a0e Merge pull request #16 from LuminosoInsight/more-tweaking 
More tweaking
 2015-07-10 14:36:18 -04:00
Andrew Lin
6efdaa308c Merge pull request #17 from LuminosoInsight/alternative-func-to-regex 
created alternate implementation of func-to-regex
 2015-07-10 14:32:49 -04:00
Joshua Chin
7706496080 added comment about parsing_range 2015-07-10 14:27:48 -04:00
Joshua Chin
447fb7aacd %c is a thing 2015-07-10 14:23:06 -04:00
Joshua Chin
2612bc23ff merge 2015-07-10 14:12:42 -04:00
Joshua Chin
87830d138b updated func_to_regex to remove end check 2015-07-10 14:10:26 -04:00
Andrew Lin
95da6985d4 Improve variable names. 2015-07-10 14:02:33 -04:00
Joshua Chin
e68f5b67a2 merged 2015-07-10 13:33:17 -04:00
Joshua Chin
7c189ef129 created alternate implementation of func-to-regex 2015-07-10 11:03:57 -04:00
Andrew Lin
6755741e7d Clarify the algorithm for range calculation using an explicit variable. 2015-07-09 16:47:33 -04:00
Andrew Lin
f591e74663 Whoops -- put back 'file' as a variable name. (The perils of trusting syntax highlighting...) 2015-07-09 16:18:56 -04:00
Andrew Lin
47ee662e24 Remove an unused temporary. 2015-07-09 15:50:44 -04:00
Andrew Lin
32b4033d63 Express the combining of word frequencies in an explicitly associative and commutative way. 2015-07-09 15:29:05 -04:00
Andrew Lin
2821f23e79 Switch to a faster 'drop everything' style cache for word_frequency(). 2015-07-09 14:32:16 -04:00
Andrew Lin
b4bae84aac Close a file a little earlier. 2015-07-09 14:03:24 -04:00
Andrew Lin
677ed6eb87 Spacing tweaks. 2015-07-09 13:26:49 -04:00
Andrew Lin
9f8464c2d1 Document the version of Unicode used to build the regexes. 2015-07-08 18:48:33 -04:00
Andrew Lin
cc6920d7e4 Tweaks to the regex generator for brevity: 
* Don't repeat the logic that generates the ranges
  * Include only unassigned characters between two accepted ranges; this causes the resulting
    regexes to be a bit more readable.
  * Rearrange the script itself to avoid long lambdas and group helper functions together
  * Precompute the list of all the character classes for speed and terseness
 2015-07-08 15:29:31 -04:00
Andrew Lin
30de0ff17f Merge pull request #6 from LuminosoInsight/code-review-leftovers 
Code review leftovers
 2015-07-07 19:39:44 -04:00
Rob Speer
41dba74da2 add docstrings and remove some brackets 2015-07-07 18:22:51 -04:00
Rob Speer
d4d7b2f72e fix duplication 2015-07-07 18:18:27 -04:00
Rob Speer
d9a5b452fd mention Google Books Ngrams in the word_frequency docstring 2015-07-07 18:07:39 -04:00
Andrew Lin
8686a47a30 Merge pull request #15 from LuminosoInsight/wordfreq-review 
General style fixes and improvements from the code review
 2015-07-07 16:53:17 -04:00
Joshua Chin
b9578ae21e removed unused imports 2015-07-07 16:21:22 -04:00
Joshua Chin
af362480d5 added documentation to load ranges 2015-07-07 16:00:37 -04:00
Joshua Chin
27ea107e6f cleaned up gen regex 2015-07-07 16:00:24 -04:00
Joshua Chin
189a5b9cd6 revert to using global mecab_tokenize variable 2015-07-07 15:47:37 -04:00
Joshua Chin
59c03e2411 updated minimum 2015-07-07 15:46:33 -04:00
Joshua Chin
f04ca8fc9e updated emoji parser 2015-07-07 15:43:34 -04:00
Joshua Chin
9b851f3afe updated docstring 2015-07-07 15:33:51 -04:00
Joshua Chin
32803b235b factored out range loading 2015-07-07 15:33:36 -04:00
Joshua Chin
ae4699029d fixed spacing 2015-07-07 15:23:15 -04:00
Joshua Chin
5510fce675 fixed gen_regex 2015-07-07 15:22:04 -04:00
Joshua Chin
f83d31a357 added arabic tests 2015-07-07 15:10:59 -04:00
Joshua Chin
9aa773aa2b changed default to minimum for word_frequency 2015-07-07 15:03:26 -04:00