139 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Joshua Chin	10b5727725	updated _emoji_char_class docstring	2015-07-07 14:58:50 -04:00
Joshua Chin	5342ea3033	removed intermediate lists	2015-07-07 14:57:45 -04:00
Joshua Chin	4b49b1a547	updated number of words to 5	2015-07-07 14:56:40 -04:00
Joshua Chin	4304a400f7	updated word_frequency docstring	2015-07-07 14:56:12 -04:00
Joshua Chin	5e8ef19321	run cB_to_freq only once per bucket	2015-07-07 14:55:13 -04:00
Joshua Chin	6a40e63060	use itertools.chain	2015-07-07 14:54:19 -04:00
Joshua Chin	bbdc064528	fixed Error string	2015-07-07 14:51:46 -04:00
Joshua Chin	22fbea4248	updated gen_regex to be run as script	2015-07-07 14:50:56 -04:00
Joshua Chin	f3f9a654ea	removed unused imports	2015-07-07 14:48:11 -04:00
Joshua Chin	f2b615b0f0	updated imports	2015-07-07 14:46:42 -04:00
Joshua Chin	b1cd2e01d3	imports are already cached	2015-07-07 14:44:50 -04:00
Joshua Chin	476a909e4d	factored out regex generation	2015-07-07 14:38:21 -04:00
Joshua Chin	781a072713	factored out emoji regex	2015-07-07 14:37:31 -04:00
Joshua Chin	ca66a5f883	updated tests	2015-07-07 14:13:28 -04:00
Rob Speer	a69ea5ad52	declare 'mecab' as an extra	2015-07-02 17:11:51 -04:00
Rob Speer	7b4ebd1805	declare that tests require mecab-python3	2015-07-02 11:29:11 -04:00
Joshua Chin	2acca8a27a	Merge pull request #14 from LuminosoInsight/add-twitter-wordlists ... Add twitter wordlists	2015-07-01 18:00:30 -04:00
Rob Speer	14cb408100	test and document new twitter wordlists	2015-07-01 17:53:38 -04:00
Rob Speer	7e3066d3fc	add Twitter-specific wordlists	2015-07-01 17:49:33 -04:00
Andrew Lin	dbc42830b4	Merge pull request #7 from LuminosoInsight/newbuild ... wordfreq 1.0b1	2015-07-01 16:51:39 -04:00
Joshua Chin	95fc0c8e9d	Merge pull request #13 from LuminosoInsight/casefold-tokens ... Case-fold instead of just lowercasing tokens	2015-07-01 11:34:02 -04:00
Rob Speer	f9a9ee7a82	update data using new build	2015-07-01 11:18:39 -04:00
Rob Speer	638467f600	case-fold instead of just lowercasing tokens	2015-06-30 15:14:02 -04:00
Rob Speer	053f372ebc	bump version number	2015-06-30 14:54:13 -04:00
Rob Speer	7d25627e43	Merge pull request #12 from LuminosoInsight/split-emoji ... Added the results of the new wordfreq_builder that splits emoji.	2015-06-30 11:32:40 -04:00
Joshua Chin	bbf7b9de34	revert changes to test_not_really_random	2015-06-30 11:29:14 -04:00
Joshua Chin	a49b66880e	changed english test to take random ascii words	2015-06-29 11:05:01 -04:00
Joshua Chin	5ed03b006c	changed japanese test because the most common japanese ascii word keeps changing	2015-06-29 11:04:19 -04:00
Joshua Chin	17f11ebd26	Japanese people do not 'lol', they 'w'	2015-06-29 11:01:13 -04:00
Joshua Chin	6f02dfc883	updated wordlists	2015-06-29 11:00:39 -04:00
Rob Speer	6c76942da2	Merge pull request #11 from LuminosoInsight/split-emoji ... wordfreq now splits emoji from text	2015-06-26 12:12:51 -04:00
Joshua Chin	811c199e15	fixed tatweel comment	2015-06-26 10:00:47 -04:00
Joshua Chin	91a14f6e6e	optimized ranges and treats unassigned codepoints like their neighbors	2015-06-25 14:38:32 -04:00
Joshua Chin	5fc448bc60	changed mecab_tokenize to a global variable	2015-06-25 13:58:30 -04:00
Joshua Chin	5cdac0c54e	added uncategorized unicodes as not punctuation	2015-06-25 13:53:54 -04:00
Joshua Chin	cebca52ea3	removes combining marks from arabic words instead of treating them as punctuation	2015-06-25 12:36:41 -04:00
Joshua Chin	83797bd276	removes arabic commas	2015-06-25 12:02:59 -04:00
Joshua Chin	6e1f7e30c6	only import mecab once	2015-06-25 11:41:19 -04:00
Joshua Chin	3bcb3e84a1	updated tests for emoji splitting	2015-06-25 11:25:51 -04:00
Joshua Chin	35a80e5f50	uses DATA_PATH instead of explicit path	2015-06-25 10:42:59 -04:00
Joshua Chin	f3a365fda9	now uses ranges	2015-06-25 10:39:04 -04:00
Joshua Chin	d10737bb51	added docstrings	2015-06-24 17:45:29 -04:00
Joshua Chin	d9ebeca734	removed duplicate of non_punct.txt	2015-06-24 17:36:57 -04:00
Joshua Chin	b198f4b0c2	added non_punct to MANIFEST.in and moved it into data	2015-06-24 17:30:01 -04:00
Joshua Chin	d372b5618c	removed old FIXME	2015-06-24 17:15:50 -04:00
Joshua Chin	f576ca58ae	caches non_punct regex in non_punct.txt	2015-06-24 17:11:50 -04:00
Rob Speer	97bbb97f63	Merge pull request #10 from LuminosoInsight/returns-none-bugfix ... word_frequency no longer returns None if it does not detect tokens	2015-06-24 17:04:36 -04:00
Joshua Chin	78c5b589c5	splits emoji from text	2015-06-24 16:50:28 -04:00
Joshua Chin	2346a0535a	word_frequency no longer returns None if it does not detect tokens	2015-06-24 14:47:26 -04:00
Joshua Chin	0cac6dfda1	Merge pull request #9 from LuminosoInsight/centibels ... Switch to a more precise centibel scale	2015-06-23 12:55:55 -04:00