Joshua Chin
449a656edd
removed TOKENIZE_TWITTER
2015-07-17 14:43:14 -04:00
Joshua Chin
00e18b7d4b
removed TOKENIZE_TWITTER option
2015-07-17 14:40:49 -04:00
Joshua Chin
772c0cddd1
more README fixes
2015-07-17 14:40:33 -04:00
Joshua Chin
0a085132f4
fixed README
2015-07-17 14:35:43 -04:00
Rob Speer
e02f41076d
Merge pull request #18 from LuminosoInsight/add-builder
...
Add wordfreq_builder as a sub-directory to wordfreq
2015-07-13 14:43:53 -04:00
Rob Speer
8633e8c2a9
update the wordfreq_builder README
2015-07-13 11:58:48 -04:00
Joshua Chin
668f0fa67f
Merge ../wordfreq_builder
2015-07-13 11:03:18 -04:00
Joshua Chin
2c573b5a0e
Merge pull request #16 from LuminosoInsight/more-tweaking
...
More tweaking
2015-07-10 14:36:18 -04:00
Andrew Lin
6efdaa308c
Merge pull request #17 from LuminosoInsight/alternative-func-to-regex
...
created alternate implementation of func-to-regex
2015-07-10 14:32:49 -04:00
Joshua Chin
7706496080
added comment about parsing_range
2015-07-10 14:27:48 -04:00
Joshua Chin
447fb7aacd
%c is a thing
2015-07-10 14:23:06 -04:00
Joshua Chin
2612bc23ff
merge
2015-07-10 14:12:42 -04:00
Joshua Chin
87830d138b
updated func_to_regex to remove end check
2015-07-10 14:10:26 -04:00
Andrew Lin
95da6985d4
Improve variable names.
2015-07-10 14:02:33 -04:00
Joshua Chin
e68f5b67a2
merged
2015-07-10 13:33:17 -04:00
Joshua Chin
7c189ef129
created alternate implementation of func-to-regex
2015-07-10 11:03:57 -04:00
Andrew Lin
6755741e7d
Clarify the algorithm for range calculation using an explicit variable.
2015-07-09 16:47:33 -04:00
Andrew Lin
f591e74663
Whoops -- put back 'file' as a variable name. (The perils of trusting syntax highlighting...)
2015-07-09 16:18:56 -04:00
Andrew Lin
47ee662e24
Remove an unused temporary.
2015-07-09 15:50:44 -04:00
Andrew Lin
32b4033d63
Express the combining of word frequencies in an explicitly associative and commutative way.
2015-07-09 15:29:05 -04:00
Andrew Lin
2821f23e79
Switch to a faster 'drop everything' style cache for word_frequency().
2015-07-09 14:32:16 -04:00
Andrew Lin
b4bae84aac
Close a file a little earlier.
2015-07-09 14:03:24 -04:00
Andrew Lin
677ed6eb87
Spacing tweaks.
2015-07-09 13:26:49 -04:00
Andrew Lin
9f8464c2d1
Document the version of Unicode used to build the regexes.
2015-07-08 18:48:33 -04:00
Andrew Lin
cc6920d7e4
Tweaks to the regex generator for brevity:
...
* Don't repeat the logic that generates the ranges
* Include only unassigned characters between two accepted ranges; this causes the resulting
regexes to be a bit more readable.
* Rearrange the script itself to avoid long lambdas and group helper functions together
* Precompute the list of all the character classes for speed and terseness
2015-07-08 15:29:31 -04:00
Andrew Lin
30de0ff17f
Merge pull request #6 from LuminosoInsight/code-review-leftovers
...
Code review leftovers
2015-07-07 19:39:44 -04:00
Rob Speer
41dba74da2
add docstrings and remove some brackets
2015-07-07 18:22:51 -04:00
Rob Speer
d4d7b2f72e
fix duplication
2015-07-07 18:18:27 -04:00
Rob Speer
d9a5b452fd
mention Google Books Ngrams in the word_frequency docstring
2015-07-07 18:07:39 -04:00
Andrew Lin
8686a47a30
Merge pull request #15 from LuminosoInsight/wordfreq-review
...
General style fixes and improvements from the code review
2015-07-07 16:53:17 -04:00
Joshua Chin
b9578ae21e
removed unused imports
2015-07-07 16:21:22 -04:00
Joshua Chin
af362480d5
added documentation to load ranges
2015-07-07 16:00:37 -04:00
Joshua Chin
27ea107e6f
cleaned up gen regex
2015-07-07 16:00:24 -04:00
Joshua Chin
189a5b9cd6
revert to using global mecab_tokenize variable
2015-07-07 15:47:37 -04:00
Joshua Chin
59c03e2411
updated minimum
2015-07-07 15:46:33 -04:00
Joshua Chin
f04ca8fc9e
updated emoji parser
2015-07-07 15:43:34 -04:00
Joshua Chin
9b851f3afe
updated docstring
2015-07-07 15:33:51 -04:00
Joshua Chin
32803b235b
factored out range loading
2015-07-07 15:33:36 -04:00
Joshua Chin
ae4699029d
fixed spacing
2015-07-07 15:23:15 -04:00
Joshua Chin
5510fce675
fixed gen_regex
2015-07-07 15:22:04 -04:00
Joshua Chin
f83d31a357
added arabic tests
2015-07-07 15:10:59 -04:00
Joshua Chin
9aa773aa2b
changed default to minimum for word_frequency
2015-07-07 15:03:26 -04:00
Joshua Chin
0b25caaf24
added docstring to top_n_list
2015-07-07 15:01:39 -04:00
Joshua Chin
bd172594d3
fix grammar
2015-07-07 14:59:28 -04:00
Joshua Chin
10b5727725
updated _emoji_char_class docstring
2015-07-07 14:58:50 -04:00
Joshua Chin
5342ea3033
removed intermediate lists
2015-07-07 14:57:45 -04:00
Joshua Chin
4b49b1a547
updated number of words to 5
2015-07-07 14:56:40 -04:00
Joshua Chin
4304a400f7
updated word_frequency docstring
2015-07-07 14:56:12 -04:00
Joshua Chin
5e8ef19321
run cB_to_freq only once per bucket
2015-07-07 14:55:13 -04:00
Joshua Chin
6a40e63060
use itertools.chain
2015-07-07 14:54:19 -04:00