iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-24 01:41:39 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
118 Commits 8 Branches 124 Tags 707 MiB
f9a9ee7a82
Commit Graph

118 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rob Speer
f9a9ee7a82 update data using new build 2015-07-01 11:18:39 -04:00
Rob Speer
638467f600 case-fold instead of just lowercasing tokens 2015-06-30 15:14:02 -04:00
Rob Speer
053f372ebc bump version number 2015-06-30 14:54:13 -04:00
Rob Speer
7d25627e43 Merge pull request #12 from LuminosoInsight/split-emoji 
Added the results of the new wordfreq_builder that splits emoji.
 2015-06-30 11:32:40 -04:00
Joshua Chin
bbf7b9de34 revert changes to test_not_really_random 2015-06-30 11:29:14 -04:00
Joshua Chin
a49b66880e changed english test to take random ascii words 2015-06-29 11:05:01 -04:00
Joshua Chin
5ed03b006c changed japanese test because the most common japanese ascii word keeps changing 2015-06-29 11:04:19 -04:00
Joshua Chin
17f11ebd26 Japanese people do not 'lol', they 'w' 2015-06-29 11:01:13 -04:00
Joshua Chin
6f02dfc883 updated wordlists 2015-06-29 11:00:39 -04:00
Rob Speer
6c76942da2 Merge pull request #11 from LuminosoInsight/split-emoji 
wordfreq now splits emoji from text
 2015-06-26 12:12:51 -04:00
Joshua Chin
811c199e15 fixed tatweel comment 2015-06-26 10:00:47 -04:00
Joshua Chin
91a14f6e6e optimized ranges and treats unassigned codepoints like their neighbors 2015-06-25 14:38:32 -04:00
Joshua Chin
5fc448bc60 changed mecab_tokenize to a global variable 2015-06-25 13:58:30 -04:00
Joshua Chin
5cdac0c54e added uncategorized unicodes as not punctuation 2015-06-25 13:53:54 -04:00
Joshua Chin
cebca52ea3 removes combining marks from arabic words instead of treating them as punctuation 2015-06-25 12:36:41 -04:00
Joshua Chin
83797bd276 removes arabic commas 2015-06-25 12:02:59 -04:00
Joshua Chin
6e1f7e30c6 only import mecab once 2015-06-25 11:41:19 -04:00
Joshua Chin
3bcb3e84a1 updated tests for emoji splitting 2015-06-25 11:25:51 -04:00
Joshua Chin
35a80e5f50 uses DATA_PATH instead of explicit path 2015-06-25 10:42:59 -04:00
Joshua Chin
f3a365fda9 now uses ranges 2015-06-25 10:39:04 -04:00
Joshua Chin
d10737bb51 added docstrings 2015-06-24 17:45:29 -04:00
Joshua Chin
d9ebeca734 removed duplicate of non_punct.txt 2015-06-24 17:36:57 -04:00
Joshua Chin
b198f4b0c2 added non_punct to MANIFEST.in and moved it into data 2015-06-24 17:30:01 -04:00
Joshua Chin
d372b5618c removed old FIXME 2015-06-24 17:15:50 -04:00
Joshua Chin
f576ca58ae caches non_punct regex in non_punct.txt 2015-06-24 17:11:50 -04:00
Rob Speer
97bbb97f63 Merge pull request #10 from LuminosoInsight/returns-none-bugfix 
word_frequency no longer returns None if it does not detect tokens
 2015-06-24 17:04:36 -04:00
Joshua Chin
78c5b589c5 splits emoji from text 2015-06-24 16:50:28 -04:00
Joshua Chin
2346a0535a word_frequency no longer returns None if it does not detect tokens 2015-06-24 14:47:26 -04:00
Joshua Chin
0cac6dfda1 Merge pull request #9 from LuminosoInsight/centibels 
Switch to a more precise centibel scale
 2015-06-23 12:55:55 -04:00
Rob Speer
d81070a13c fix redundant mention of the header 2015-06-23 12:53:57 -04:00
Rob Speer
7862a4d2b6 Switch to a more precise centibel scale. 2015-06-22 17:36:30 -04:00
Rob Speer
28c9edf135 Update the copyright year in the license 
I don't know when you need to update the year, but the code has substantially changed since 2013.
 2015-06-18 18:55:59 -04:00
Rob Speer
cd3270168f Rename MIT-LICENSE to MIT-LICENSE.txt 
This makes the filename match what README.txt refers to.
 2015-06-18 18:55:18 -04:00
Joshua Chin
90fc3970c6 updated data with a clean build 2015-06-18 11:38:57 -04:00
Joshua Chin
35f472fcf9 updated test because the new tokenizer removes URLs 2015-06-18 11:38:28 -04:00
Rob Speer
981cc249f1 copy-edit some docstrings 2015-06-17 14:42:38 -04:00
Rob Speer
13988f8e3d Merge pull request #8 from LuminosoInsight/newbuild-refactor 
Refactored the newbuild branch, in response to the preliminary review notes
 2015-06-17 14:35:37 -04:00
Joshua Chin
68b1c121bd updated dB_to_freq docstring 2015-06-17 14:34:03 -04:00
Joshua Chin
a289ab7f8b clearified the docstrings for random_words and random_ascii_words 2015-06-17 14:26:06 -04:00
Joshua Chin
9f288bac31 corrected available_languages to return a dict of strs to strs 2015-06-17 12:43:13 -04:00
Joshua Chin
6cc962bfea changed yield to yield from in iter_wordlist 2015-06-17 12:38:31 -04:00
Joshua Chin
9b30da4dec updated db_to_freq docstring 2015-06-17 12:24:23 -04:00
Joshua Chin
7e808bf7c1 added docstrings 2015-06-17 12:20:50 -04:00
Joshua Chin
053e4da3e6 removed temporary variable 2015-06-17 12:16:02 -04:00
Rob Speer
ed19d79c5a clearer error on py2 2015-05-28 14:05:11 -04:00
Rob Speer
0f4ca80026 add installation instructions to the readme 2015-05-28 14:02:12 -04:00
Rob Speer
611a6a35de update Japanese data; test Japanese and token combining 2015-05-28 14:01:56 -04:00
Rob Speer
05cf94d1fd Work on making Japanese tokenization use MeCab consistently 2015-05-27 18:10:25 -04:00
Rob Speer
0e5156e162 Merge branch 'master' into newbuild 
Conflicts:
	setup.py
	wordfreq/build.py
	wordfreq/config.py
 2015-05-21 20:41:47 -04:00
Rob Speer
84e5edcea1 rebuild data 2015-05-21 20:36:15 -04:00