iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-27 02:48:51 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
453 Commits 8 Branches 124 Tags 707 MiB
afe6537994
Commit Graph

453 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Joshua Chin
22fbea4248 updated gen_regex to be run as script 2015-07-07 14:50:56 -04:00
Joshua Chin
f3f9a654ea removed unused imports 2015-07-07 14:48:11 -04:00
Joshua Chin
f2b615b0f0 updated imports 2015-07-07 14:46:42 -04:00
Joshua Chin
b1cd2e01d3 imports are already cached 2015-07-07 14:44:50 -04:00
Joshua Chin
476a909e4d factored out regex generation 2015-07-07 14:38:21 -04:00
Joshua Chin
781a072713 factored out emoji regex 2015-07-07 14:37:31 -04:00
Joshua Chin
ca66a5f883 updated tests 2015-07-07 14:13:28 -04:00
Joshua Chin
b0f759d322 Removes mention of Rosette from README 2015-07-07 10:32:16 -04:00
Rob Speer
a69ea5ad52 declare 'mecab' as an extra 2015-07-02 17:11:51 -04:00
Rob Speer
7b4ebd1805 declare that tests require mecab-python3 2015-07-02 11:29:11 -04:00
Joshua Chin
25561b994f Merge pull request #5 from LuminosoInsight/add-twitter-build 
add 'twitter' as a final build, and a new build dir
 2015-07-01 18:01:32 -04:00
Joshua Chin
2acca8a27a Merge pull request #14 from LuminosoInsight/add-twitter-wordlists 
Add twitter wordlists
 2015-07-01 18:00:30 -04:00
Rob Speer
14cb408100 test and document new twitter wordlists 2015-07-01 17:53:38 -04:00
Rob Speer
7e3066d3fc add Twitter-specific wordlists 2015-07-01 17:49:33 -04:00
Rob Speer
10c04d116f add 'twitter' as a final build, and a new build dir 
The `data/dist` directory is now a convenient place to find the final
built files that can be copied into wordfreq.
 2015-07-01 17:45:39 -04:00
Andrew Lin
dbc42830b4 Merge pull request #7 from LuminosoInsight/newbuild 
wordfreq 1.0b1
 2015-07-01 16:51:39 -04:00
Joshua Chin
d560b7c42b Merge pull request #4 from LuminosoInsight/tokenization-cleanup 
remove wiki2tokens and tokenize_wikipedia
 2015-07-01 11:34:30 -04:00
Joshua Chin
95fc0c8e9d Merge pull request #13 from LuminosoInsight/casefold-tokens 
Case-fold instead of just lowercasing tokens
 2015-07-01 11:34:02 -04:00
Rob Speer
f9a9ee7a82 update data using new build 2015-07-01 11:18:39 -04:00
Rob Speer
37375383e8 cope with occasional Unicode errors in the input 2015-06-30 17:05:40 -04:00
Rob Speer
4771c12814 remove wiki2tokens and tokenize_wikipedia 
These components are no longer necessary. Wikipedia output can and
should be tokenized with the standard tokenizer, instead of the
almost-equivalent one in the Nim code.
 2015-06-30 15:28:01 -04:00
Rob Speer
9a2855394d fix comment and whitespace involving tokenize_twitter 2015-06-30 15:18:37 -04:00
Rob Speer
638467f600 case-fold instead of just lowercasing tokens 2015-06-30 15:14:02 -04:00
Rob Speer
053f372ebc bump version number 2015-06-30 14:54:13 -04:00
Rob Speer
7d25627e43 Merge pull request #12 from LuminosoInsight/split-emoji 
Added the results of the new wordfreq_builder that splits emoji.
 2015-06-30 11:32:40 -04:00
Joshua Chin
bbf7b9de34 revert changes to test_not_really_random 2015-06-30 11:29:14 -04:00
Joshua Chin
a49b66880e changed english test to take random ascii words 2015-06-29 11:05:01 -04:00
Joshua Chin
5ed03b006c changed japanese test because the most common japanese ascii word keeps changing 2015-06-29 11:04:19 -04:00
Joshua Chin
17f11ebd26 Japanese people do not 'lol', they 'w' 2015-06-29 11:01:13 -04:00
Joshua Chin
6f02dfc883 updated wordlists 2015-06-29 11:00:39 -04:00
Rob Speer
6c76942da2 Merge pull request #11 from LuminosoInsight/split-emoji 
wordfreq now splits emoji from text
 2015-06-26 12:12:51 -04:00
Joshua Chin
811c199e15 fixed tatweel comment 2015-06-26 10:00:47 -04:00
Joshua Chin
91a14f6e6e optimized ranges and treats unassigned codepoints like their neighbors 2015-06-25 14:38:32 -04:00
Joshua Chin
5fc448bc60 changed mecab_tokenize to a global variable 2015-06-25 13:58:30 -04:00
Joshua Chin
5cdac0c54e added uncategorized unicodes as not punctuation 2015-06-25 13:53:54 -04:00
Joshua Chin
cebca52ea3 removes combining marks from arabic words instead of treating them as punctuation 2015-06-25 12:36:41 -04:00
Joshua Chin
83797bd276 removes arabic commas 2015-06-25 12:02:59 -04:00
Joshua Chin
6e1f7e30c6 only import mecab once 2015-06-25 11:41:19 -04:00
Joshua Chin
3bcb3e84a1 updated tests for emoji splitting 2015-06-25 11:25:51 -04:00
Joshua Chin
35a80e5f50 uses DATA_PATH instead of explicit path 2015-06-25 10:42:59 -04:00
Joshua Chin
f3a365fda9 now uses ranges 2015-06-25 10:39:04 -04:00
Joshua Chin
d10737bb51 added docstrings 2015-06-24 17:45:29 -04:00
Joshua Chin
d9ebeca734 removed duplicate of non_punct.txt 2015-06-24 17:36:57 -04:00
Joshua Chin
b198f4b0c2 added non_punct to MANIFEST.in and moved it into data 2015-06-24 17:30:01 -04:00
Joshua Chin
d372b5618c removed old FIXME 2015-06-24 17:15:50 -04:00
Joshua Chin
f576ca58ae caches non_punct regex in non_punct.txt 2015-06-24 17:11:50 -04:00
Rob Speer
97bbb97f63 Merge pull request #10 from LuminosoInsight/returns-none-bugfix 
word_frequency no longer returns None if it does not detect tokens
 2015-06-24 17:04:36 -04:00
Joshua Chin
78c5b589c5 splits emoji from text 2015-06-24 16:50:28 -04:00
Joshua Chin
2346a0535a word_frequency no longer returns None if it does not detect tokens 2015-06-24 14:47:26 -04:00
Joshua Chin
a59070479e Merge pull request #3 from LuminosoInsight/centibels 
Switch to a centibel scale, add a header to the data
 2015-06-23 12:59:20 -04:00