iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-26 10:28:52 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
332 Commits 8 Branches 124 Tags 707 MiB
2d58ba94f2
Commit Graph

332 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Lin
677ed6eb87 Spacing tweaks. 2015-07-09 13:26:49 -04:00
Andrew Lin
9f8464c2d1 Document the version of Unicode used to build the regexes. 2015-07-08 18:48:33 -04:00
Andrew Lin
cc6920d7e4 Tweaks to the regex generator for brevity: 
* Don't repeat the logic that generates the ranges
  * Include only unassigned characters between two accepted ranges; this causes the resulting
    regexes to be a bit more readable.
  * Rearrange the script itself to avoid long lambdas and group helper functions together
  * Precompute the list of all the character classes for speed and terseness
 2015-07-08 15:29:31 -04:00
Andrew Lin
30de0ff17f Merge pull request #6 from LuminosoInsight/code-review-leftovers 
Code review leftovers
 2015-07-07 19:39:44 -04:00
Rob Speer
41dba74da2 add docstrings and remove some brackets 2015-07-07 18:22:51 -04:00
Rob Speer
d4d7b2f72e fix duplication 2015-07-07 18:18:27 -04:00
Rob Speer
d9a5b452fd mention Google Books Ngrams in the word_frequency docstring 2015-07-07 18:07:39 -04:00
Andrew Lin
8686a47a30 Merge pull request #15 from LuminosoInsight/wordfreq-review 
General style fixes and improvements from the code review
 2015-07-07 16:53:17 -04:00
Joshua Chin
b9578ae21e removed unused imports 2015-07-07 16:21:22 -04:00
Joshua Chin
af362480d5 added documentation to load ranges 2015-07-07 16:00:37 -04:00
Joshua Chin
27ea107e6f cleaned up gen regex 2015-07-07 16:00:24 -04:00
Joshua Chin
189a5b9cd6 revert to using global mecab_tokenize variable 2015-07-07 15:47:37 -04:00
Joshua Chin
59c03e2411 updated minimum 2015-07-07 15:46:33 -04:00
Joshua Chin
f04ca8fc9e updated emoji parser 2015-07-07 15:43:34 -04:00
Joshua Chin
9b851f3afe updated docstring 2015-07-07 15:33:51 -04:00
Joshua Chin
32803b235b factored out range loading 2015-07-07 15:33:36 -04:00
Joshua Chin
ae4699029d fixed spacing 2015-07-07 15:23:15 -04:00
Joshua Chin
5510fce675 fixed gen_regex 2015-07-07 15:22:04 -04:00
Joshua Chin
f83d31a357 added arabic tests 2015-07-07 15:10:59 -04:00
Joshua Chin
9aa773aa2b changed default to minimum for word_frequency 2015-07-07 15:03:26 -04:00
Joshua Chin
0b25caaf24 added docstring to top_n_list 2015-07-07 15:01:39 -04:00
Joshua Chin
bd172594d3 fix grammar 2015-07-07 14:59:28 -04:00
Joshua Chin
10b5727725 updated _emoji_char_class docstring 2015-07-07 14:58:50 -04:00
Joshua Chin
5342ea3033 removed intermediate lists 2015-07-07 14:57:45 -04:00
Joshua Chin
4b49b1a547 updated number of words to 5 2015-07-07 14:56:40 -04:00
Joshua Chin
4304a400f7 updated word_frequency docstring 2015-07-07 14:56:12 -04:00
Joshua Chin
5e8ef19321 run cB_to_freq only once per bucket 2015-07-07 14:55:13 -04:00
Joshua Chin
6a40e63060 use itertools.chain 2015-07-07 14:54:19 -04:00
Joshua Chin
bbdc064528 fixed Error string 2015-07-07 14:51:46 -04:00
Joshua Chin
22fbea4248 updated gen_regex to be run as script 2015-07-07 14:50:56 -04:00
Joshua Chin
f3f9a654ea removed unused imports 2015-07-07 14:48:11 -04:00
Joshua Chin
f2b615b0f0 updated imports 2015-07-07 14:46:42 -04:00
Joshua Chin
b1cd2e01d3 imports are already cached 2015-07-07 14:44:50 -04:00
Joshua Chin
476a909e4d factored out regex generation 2015-07-07 14:38:21 -04:00
Joshua Chin
781a072713 factored out emoji regex 2015-07-07 14:37:31 -04:00
Joshua Chin
ca66a5f883 updated tests 2015-07-07 14:13:28 -04:00
Joshua Chin
b0f759d322 Removes mention of Rosette from README 2015-07-07 10:32:16 -04:00
Rob Speer
a69ea5ad52 declare 'mecab' as an extra 2015-07-02 17:11:51 -04:00
Rob Speer
7b4ebd1805 declare that tests require mecab-python3 2015-07-02 11:29:11 -04:00
Joshua Chin
25561b994f Merge pull request #5 from LuminosoInsight/add-twitter-build 
add 'twitter' as a final build, and a new build dir
 2015-07-01 18:01:32 -04:00
Joshua Chin
2acca8a27a Merge pull request #14 from LuminosoInsight/add-twitter-wordlists 
Add twitter wordlists
 2015-07-01 18:00:30 -04:00
Rob Speer
14cb408100 test and document new twitter wordlists 2015-07-01 17:53:38 -04:00
Rob Speer
7e3066d3fc add Twitter-specific wordlists 2015-07-01 17:49:33 -04:00
Rob Speer
10c04d116f add 'twitter' as a final build, and a new build dir 
The `data/dist` directory is now a convenient place to find the final
built files that can be copied into wordfreq.
 2015-07-01 17:45:39 -04:00
Andrew Lin
dbc42830b4 Merge pull request #7 from LuminosoInsight/newbuild 
wordfreq 1.0b1
 2015-07-01 16:51:39 -04:00
Joshua Chin
d560b7c42b Merge pull request #4 from LuminosoInsight/tokenization-cleanup 
remove wiki2tokens and tokenize_wikipedia
 2015-07-01 11:34:30 -04:00
Joshua Chin
95fc0c8e9d Merge pull request #13 from LuminosoInsight/casefold-tokens 
Case-fold instead of just lowercasing tokens
 2015-07-01 11:34:02 -04:00
Rob Speer
f9a9ee7a82 update data using new build 2015-07-01 11:18:39 -04:00
Rob Speer
37375383e8 cope with occasional Unicode errors in the input 2015-06-30 17:05:40 -04:00
Rob Speer
4771c12814 remove wiki2tokens and tokenize_wikipedia 
These components are no longer necessary. Wikipedia output can and
should be tokenized with the standard tokenizer, instead of the
almost-equivalent one in the Nim code.
 2015-06-30 15:28:01 -04:00