Rob Speer
f9a9ee7a82
update data using new build
2015-07-01 11:18:39 -04:00
Rob Speer
37375383e8
cope with occasional Unicode errors in the input
2015-06-30 17:05:40 -04:00
Rob Speer
4771c12814
remove wiki2tokens and tokenize_wikipedia
...
These components are no longer necessary. Wikipedia output can and
should be tokenized with the standard tokenizer, instead of the
almost-equivalent one in the Nim code.
2015-06-30 15:28:01 -04:00
Rob Speer
9a2855394d
fix comment and whitespace involving tokenize_twitter
2015-06-30 15:18:37 -04:00
Rob Speer
638467f600
case-fold instead of just lowercasing tokens
2015-06-30 15:14:02 -04:00
Rob Speer
053f372ebc
bump version number
2015-06-30 14:54:13 -04:00
Rob Speer
7d25627e43
Merge pull request #12 from LuminosoInsight/split-emoji
...
Added the results of the new wordfreq_builder that splits emoji.
2015-06-30 11:32:40 -04:00
Joshua Chin
bbf7b9de34
revert changes to test_not_really_random
2015-06-30 11:29:14 -04:00
Joshua Chin
a49b66880e
changed english test to take random ascii words
2015-06-29 11:05:01 -04:00
Joshua Chin
5ed03b006c
changed japanese test because the most common japanese ascii word keeps changing
2015-06-29 11:04:19 -04:00
Joshua Chin
17f11ebd26
Japanese people do not 'lol', they 'w'
2015-06-29 11:01:13 -04:00
Joshua Chin
6f02dfc883
updated wordlists
2015-06-29 11:00:39 -04:00
Rob Speer
6c76942da2
Merge pull request #11 from LuminosoInsight/split-emoji
...
wordfreq now splits emoji from text
2015-06-26 12:12:51 -04:00
Joshua Chin
811c199e15
fixed tatweel comment
2015-06-26 10:00:47 -04:00
Joshua Chin
91a14f6e6e
optimized ranges and treats unassigned codepoints like their neighbors
2015-06-25 14:38:32 -04:00
Joshua Chin
5fc448bc60
changed mecab_tokenize to a global variable
2015-06-25 13:58:30 -04:00
Joshua Chin
5cdac0c54e
added uncategorized unicodes as not punctuation
2015-06-25 13:53:54 -04:00
Joshua Chin
cebca52ea3
removes combining marks from arabic words instead of treating them as punctuation
2015-06-25 12:36:41 -04:00
Joshua Chin
83797bd276
removes arabic commas
2015-06-25 12:02:59 -04:00
Joshua Chin
6e1f7e30c6
only import mecab once
2015-06-25 11:41:19 -04:00
Joshua Chin
3bcb3e84a1
updated tests for emoji splitting
2015-06-25 11:25:51 -04:00
Joshua Chin
35a80e5f50
uses DATA_PATH instead of explicit path
2015-06-25 10:42:59 -04:00
Joshua Chin
f3a365fda9
now uses ranges
2015-06-25 10:39:04 -04:00
Joshua Chin
d10737bb51
added docstrings
2015-06-24 17:45:29 -04:00
Joshua Chin
d9ebeca734
removed duplicate of non_punct.txt
2015-06-24 17:36:57 -04:00
Joshua Chin
b198f4b0c2
added non_punct to MANIFEST.in and moved it into data
2015-06-24 17:30:01 -04:00
Joshua Chin
d372b5618c
removed old FIXME
2015-06-24 17:15:50 -04:00
Joshua Chin
f576ca58ae
caches non_punct regex in non_punct.txt
2015-06-24 17:11:50 -04:00
Rob Speer
97bbb97f63
Merge pull request #10 from LuminosoInsight/returns-none-bugfix
...
word_frequency no longer returns None if it does not detect tokens
2015-06-24 17:04:36 -04:00
Joshua Chin
78c5b589c5
splits emoji from text
2015-06-24 16:50:28 -04:00
Joshua Chin
2346a0535a
word_frequency no longer returns None if it does not detect tokens
2015-06-24 14:47:26 -04:00
Joshua Chin
a59070479e
Merge pull request #3 from LuminosoInsight/centibels
...
Switch to a centibel scale, add a header to the data
2015-06-23 12:59:20 -04:00
Joshua Chin
0cac6dfda1
Merge pull request #9 from LuminosoInsight/centibels
...
Switch to a more precise centibel scale
2015-06-23 12:55:55 -04:00
Rob Speer
d81070a13c
fix redundant mention of the header
2015-06-23 12:53:57 -04:00
Rob Speer
f305679caf
Switch to a centibel scale, add a header to the data
2015-06-22 17:38:13 -04:00
Rob Speer
7862a4d2b6
Switch to a more precise centibel scale.
2015-06-22 17:36:30 -04:00
Rob Speer
d16683f2b9
Merge pull request #2 from LuminosoInsight/review-refactor
...
Adds a number of bugfixes and improvements to wordfreq_builder
2015-06-19 15:29:52 -04:00
Rob Speer
5bc1f0c097
restore missing Russian OpenSubtitles data
2015-06-19 12:36:08 -04:00
Rob Speer
28c9edf135
Update the copyright year in the license
...
I don't know when you need to update the year, but the code has substantially changed since 2013.
2015-06-18 18:55:59 -04:00
Rob Speer
cd3270168f
Rename MIT-LICENSE to MIT-LICENSE.txt
...
This makes the filename match what README.txt refers to.
2015-06-18 18:55:18 -04:00
Joshua Chin
90fc3970c6
updated data with a clean build
2015-06-18 11:38:57 -04:00
Joshua Chin
35f472fcf9
updated test because the new tokenizer removes URLs
2015-06-18 11:38:28 -04:00
Joshua Chin
3746af1350
updated freqs_to_dBpack docstring
2015-06-18 10:32:53 -04:00
Joshua Chin
59ce14cdd0
revised read_freqs docstring
2015-06-18 10:28:22 -04:00
Joshua Chin
04bf6aadcc
updated monolingual_tokenize_file docstring, and removed unused argument
2015-06-18 10:20:54 -04:00
Joshua Chin
91dd73a2b5
tokenize_file should ignore lines with unknown languages
2015-06-18 10:18:57 -04:00
Joshua Chin
ffc01c75a0
Fixed CLD2_BAD_CHAR regex
2015-06-18 10:18:00 -04:00
Rob Speer
981cc249f1
copy-edit some docstrings
2015-06-17 14:42:38 -04:00
Rob Speer
13988f8e3d
Merge pull request #8 from LuminosoInsight/newbuild-refactor
...
Refactored the newbuild branch, in response to the preliminary review notes
2015-06-17 14:35:37 -04:00
Joshua Chin
68b1c121bd
updated dB_to_freq docstring
2015-06-17 14:34:03 -04:00
