316 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Joshua Chin	6e1f7e30c6	only import mecab once	2015-06-25 11:41:19 -04:00
Joshua Chin	3bcb3e84a1	updated tests for emoji splitting	2015-06-25 11:25:51 -04:00
Joshua Chin	35a80e5f50	uses DATA_PATH instead of explicit path	2015-06-25 10:42:59 -04:00
Joshua Chin	f3a365fda9	now uses ranges	2015-06-25 10:39:04 -04:00
Joshua Chin	d10737bb51	added docstrings	2015-06-24 17:45:29 -04:00
Joshua Chin	d9ebeca734	removed duplicate of non_punct.txt	2015-06-24 17:36:57 -04:00
Joshua Chin	b198f4b0c2	added non_punct to MANIFEST.in and moved it into data	2015-06-24 17:30:01 -04:00
Joshua Chin	d372b5618c	removed old FIXME	2015-06-24 17:15:50 -04:00
Joshua Chin	f576ca58ae	caches non_punct regex in non_punct.txt	2015-06-24 17:11:50 -04:00
Rob Speer	97bbb97f63	Merge pull request #10 from LuminosoInsight/returns-none-bugfix ... word_frequency no longer returns None if it does not detect tokens	2015-06-24 17:04:36 -04:00
Joshua Chin	78c5b589c5	splits emoji from text	2015-06-24 16:50:28 -04:00
Joshua Chin	2346a0535a	word_frequency no longer returns None if it does not detect tokens	2015-06-24 14:47:26 -04:00
Joshua Chin	a59070479e	Merge pull request #3 from LuminosoInsight/centibels ... Switch to a centibel scale, add a header to the data	2015-06-23 12:59:20 -04:00
Joshua Chin	0cac6dfda1	Merge pull request #9 from LuminosoInsight/centibels ... Switch to a more precise centibel scale	2015-06-23 12:55:55 -04:00
Rob Speer	d81070a13c	fix redundant mention of the header	2015-06-23 12:53:57 -04:00
Rob Speer	f305679caf	Switch to a centibel scale, add a header to the data	2015-06-22 17:38:13 -04:00
Rob Speer	7862a4d2b6	Switch to a more precise centibel scale.	2015-06-22 17:36:30 -04:00
Rob Speer	d16683f2b9	Merge pull request #2 from LuminosoInsight/review-refactor ... Adds a number of bugfixes and improvements to wordfreq_builder	2015-06-19 15:29:52 -04:00
Rob Speer	5bc1f0c097	restore missing Russian OpenSubtitles data	2015-06-19 12:36:08 -04:00
Rob Speer	28c9edf135	Update the copyright year in the license ... I don't know when you need to update the year, but the code has substantially changed since 2013.	2015-06-18 18:55:59 -04:00
Rob Speer	cd3270168f	Rename MIT-LICENSE to MIT-LICENSE.txt ... This makes the filename match what README.txt refers to.	2015-06-18 18:55:18 -04:00
Joshua Chin	90fc3970c6	updated data with a clean build	2015-06-18 11:38:57 -04:00
Joshua Chin	35f472fcf9	updated test because the new tokenizer removes URLs	2015-06-18 11:38:28 -04:00
Joshua Chin	3746af1350	updated freqs_to_dBpack docstring	2015-06-18 10:32:53 -04:00
Joshua Chin	59ce14cdd0	revised read_freqs docstring	2015-06-18 10:28:22 -04:00
Joshua Chin	04bf6aadcc	updated monolingual_tokenize_file docstring, and removed unused argument	2015-06-18 10:20:54 -04:00
Joshua Chin	91dd73a2b5	tokenize_file should ignore lines with unknown languages	2015-06-18 10:18:57 -04:00
Joshua Chin	ffc01c75a0	Fixed CLD2_BAD_CHAR regex	2015-06-18 10:18:00 -04:00
Rob Speer	981cc249f1	copy-edit some docstrings	2015-06-17 14:42:38 -04:00
Rob Speer	13988f8e3d	Merge pull request #8 from LuminosoInsight/newbuild-refactor ... Refactored the newbuild branch, in response to the preliminary review notes	2015-06-17 14:35:37 -04:00
Joshua Chin	68b1c121bd	updated dB_to_freq docstring	2015-06-17 14:34:03 -04:00
Joshua Chin	a289ab7f8b	clearified the docstrings for random_words and random_ascii_words	2015-06-17 14:26:06 -04:00
Joshua Chin	8277de2c7f	changed tokenize_file: cld2 return 'un' instead of None if it cannot recognize the language	2015-06-17 14:19:28 -04:00
Joshua Chin	b24f31d30a	tokenize_file: don't join tokens if language is None	2015-06-17 14:18:18 -04:00
Joshua Chin	9f288bac31	corrected available_languages to return a dict of strs to strs	2015-06-17 12:43:13 -04:00
Joshua Chin	6cc962bfea	changed yield to yield from in iter_wordlist	2015-06-17 12:38:31 -04:00
Joshua Chin	9b30da4dec	updated db_to_freq docstring	2015-06-17 12:24:23 -04:00
Joshua Chin	7e808bf7c1	added docstrings	2015-06-17 12:20:50 -04:00
Joshua Chin	053e4da3e6	removed temporary variable	2015-06-17 12:16:02 -04:00
Joshua Chin	99d97956e6	automatically closes input file in tokenize_file	2015-06-17 11:42:34 -04:00
Rob Speer	855da9974d	Merge pull request #1 from LuminosoInsight/cld2-tokenizer ... Replaced usage of Rosette with the CLD2 language recognizer and the wordfreq tokenizer	2015-06-17 11:33:15 -04:00
Joshua Chin	e50c0c6917	updated test to check number parsing	2015-06-17 11:30:25 -04:00
Joshua Chin	c71e93611b	fixed build process	2015-06-17 11:25:07 -04:00
Joshua Chin	8317ea6d51	updated directory of twitter output	2015-06-16 17:32:58 -04:00
Joshua Chin	da93bc89c2	removed intermediate twitter file rules	2015-06-16 17:28:09 -04:00
Joshua Chin	87f08780c8	improved tokenize_file and updated docstring	2015-06-16 17:27:27 -04:00
Joshua Chin	bea8963a79	renamed pretokenize_twitter to tokenize twitter, and deleted format_twitter	2015-06-16 17:26:52 -04:00
Joshua Chin	aeedb408b7	fixed bugs and removed unused code	2015-06-16 17:25:06 -04:00
Joshua Chin	64644d8ede	changed tokenizer to only strip t.co urls	2015-06-16 16:11:31 -04:00
Joshua Chin	b649d45e61	Added codepoints U+10FFFE and U+10FFFF to CLD2_BAD_CHAR_RANGE	2015-06-16 16:03:58 -04:00