iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-24 09:51:38 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
418 Commits 8 Branches 124 Tags 707 MiB
4ec6b56faa
Commit Graph

418 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Joshua Chin
cebca52ea3 removes combining marks from arabic words instead of treating them as punctuation 2015-06-25 12:36:41 -04:00
Joshua Chin
83797bd276 removes arabic commas 2015-06-25 12:02:59 -04:00
Joshua Chin
6e1f7e30c6 only import mecab once 2015-06-25 11:41:19 -04:00
Joshua Chin
3bcb3e84a1 updated tests for emoji splitting 2015-06-25 11:25:51 -04:00
Joshua Chin
35a80e5f50 uses DATA_PATH instead of explicit path 2015-06-25 10:42:59 -04:00
Joshua Chin
f3a365fda9 now uses ranges 2015-06-25 10:39:04 -04:00
Joshua Chin
d10737bb51 added docstrings 2015-06-24 17:45:29 -04:00
Joshua Chin
d9ebeca734 removed duplicate of non_punct.txt 2015-06-24 17:36:57 -04:00
Joshua Chin
b198f4b0c2 added non_punct to MANIFEST.in and moved it into data 2015-06-24 17:30:01 -04:00
Joshua Chin
d372b5618c removed old FIXME 2015-06-24 17:15:50 -04:00
Joshua Chin
f576ca58ae caches non_punct regex in non_punct.txt 2015-06-24 17:11:50 -04:00
Rob Speer
97bbb97f63 Merge pull request #10 from LuminosoInsight/returns-none-bugfix 
word_frequency no longer returns None if it does not detect tokens
 2015-06-24 17:04:36 -04:00
Joshua Chin
78c5b589c5 splits emoji from text 2015-06-24 16:50:28 -04:00
Joshua Chin
2346a0535a word_frequency no longer returns None if it does not detect tokens 2015-06-24 14:47:26 -04:00
Joshua Chin
a59070479e Merge pull request #3 from LuminosoInsight/centibels 
Switch to a centibel scale, add a header to the data
 2015-06-23 12:59:20 -04:00
Joshua Chin
0cac6dfda1 Merge pull request #9 from LuminosoInsight/centibels 
Switch to a more precise centibel scale
 2015-06-23 12:55:55 -04:00
Rob Speer
d81070a13c fix redundant mention of the header 2015-06-23 12:53:57 -04:00
Rob Speer
f305679caf Switch to a centibel scale, add a header to the data 2015-06-22 17:38:13 -04:00
Rob Speer
7862a4d2b6 Switch to a more precise centibel scale. 2015-06-22 17:36:30 -04:00
Rob Speer
d16683f2b9 Merge pull request #2 from LuminosoInsight/review-refactor 
Adds a number of bugfixes and improvements to wordfreq_builder
 2015-06-19 15:29:52 -04:00
Rob Speer
5bc1f0c097 restore missing Russian OpenSubtitles data 2015-06-19 12:36:08 -04:00
Rob Speer
28c9edf135 Update the copyright year in the license 
I don't know when you need to update the year, but the code has substantially changed since 2013.
 2015-06-18 18:55:59 -04:00
Rob Speer
cd3270168f Rename MIT-LICENSE to MIT-LICENSE.txt 
This makes the filename match what README.txt refers to.
 2015-06-18 18:55:18 -04:00
Joshua Chin
90fc3970c6 updated data with a clean build 2015-06-18 11:38:57 -04:00
Joshua Chin
35f472fcf9 updated test because the new tokenizer removes URLs 2015-06-18 11:38:28 -04:00
Joshua Chin
3746af1350 updated freqs_to_dBpack docstring 2015-06-18 10:32:53 -04:00
Joshua Chin
59ce14cdd0 revised read_freqs docstring 2015-06-18 10:28:22 -04:00
Joshua Chin
04bf6aadcc updated monolingual_tokenize_file docstring, and removed unused argument 2015-06-18 10:20:54 -04:00
Joshua Chin
91dd73a2b5 tokenize_file should ignore lines with unknown languages 2015-06-18 10:18:57 -04:00
Joshua Chin
ffc01c75a0 Fixed CLD2_BAD_CHAR regex 2015-06-18 10:18:00 -04:00
Rob Speer
981cc249f1 copy-edit some docstrings 2015-06-17 14:42:38 -04:00
Rob Speer
13988f8e3d Merge pull request #8 from LuminosoInsight/newbuild-refactor 
Refactored the newbuild branch, in response to the preliminary review notes
 2015-06-17 14:35:37 -04:00
Joshua Chin
68b1c121bd updated dB_to_freq docstring 2015-06-17 14:34:03 -04:00
Joshua Chin
a289ab7f8b clearified the docstrings for random_words and random_ascii_words 2015-06-17 14:26:06 -04:00
Joshua Chin
8277de2c7f changed tokenize_file: cld2 return 'un' instead of None if it cannot recognize the language 2015-06-17 14:19:28 -04:00
Joshua Chin
b24f31d30a tokenize_file: don't join tokens if language is None 2015-06-17 14:18:18 -04:00
Joshua Chin
9f288bac31 corrected available_languages to return a dict of strs to strs 2015-06-17 12:43:13 -04:00
Joshua Chin
6cc962bfea changed yield to yield from in iter_wordlist 2015-06-17 12:38:31 -04:00
Joshua Chin
9b30da4dec updated db_to_freq docstring 2015-06-17 12:24:23 -04:00
Joshua Chin
7e808bf7c1 added docstrings 2015-06-17 12:20:50 -04:00
Joshua Chin
053e4da3e6 removed temporary variable 2015-06-17 12:16:02 -04:00
Joshua Chin
99d97956e6 automatically closes input file in tokenize_file 2015-06-17 11:42:34 -04:00
Rob Speer
855da9974d Merge pull request #1 from LuminosoInsight/cld2-tokenizer 
Replaced usage of Rosette with the CLD2 language recognizer and the wordfreq tokenizer
 2015-06-17 11:33:15 -04:00
Joshua Chin
e50c0c6917 updated test to check number parsing 2015-06-17 11:30:25 -04:00
Joshua Chin
c71e93611b fixed build process 2015-06-17 11:25:07 -04:00
Joshua Chin
8317ea6d51 updated directory of twitter output 2015-06-16 17:32:58 -04:00
Joshua Chin
da93bc89c2 removed intermediate twitter file rules 2015-06-16 17:28:09 -04:00
Joshua Chin
87f08780c8 improved tokenize_file and updated docstring 2015-06-16 17:27:27 -04:00
Joshua Chin
bea8963a79 renamed pretokenize_twitter to tokenize twitter, and deleted format_twitter 2015-06-16 17:26:52 -04:00
Joshua Chin
aeedb408b7 fixed bugs and removed unused code 2015-06-16 17:25:06 -04:00