11 Commits

Author	SHA1	Message	Date
Rob Speer	3bf59fec57	test and document new twitter wordlists ... Former-commit-id: 14cb408100	2015-07-01 17:53:38 -04:00
Rob Speer	b84ba2bc2e	update data using new build ... Former-commit-id: f9a9ee7a82	2015-07-01 11:18:39 -04:00
Rob Speer	8cac81666a	case-fold instead of just lowercasing tokens ... Former-commit-id: 638467f600	2015-06-30 15:14:02 -04:00
Joshua Chin	5cc3dce834	revert changes to test_not_really_random ... Former-commit-id: bbf7b9de34	2015-06-30 11:29:14 -04:00
Joshua Chin	53c558ca90	changed english test to take random ascii words ... Former-commit-id: a49b66880e	2015-06-29 11:05:01 -04:00
Joshua Chin	ea5470a85a	changed japanese test because the most common japanese ascii word keeps changing ... Former-commit-id: 5ed03b006c	2015-06-29 11:04:19 -04:00
Joshua Chin	000491c7cc	Japanese people do not 'lol', they 'w' ... Former-commit-id: 17f11ebd26	2015-06-29 11:01:13 -04:00
Joshua Chin	09966989fb	updated tests for emoji splitting ... Former-commit-id: 3bcb3e84a1	2015-06-25 11:25:51 -04:00
Rob Speer	b4600c9bd1	Switch to a more precise centibel scale. ... Former-commit-id: 7862a4d2b6	2015-06-22 17:36:30 -04:00
Joshua Chin	529aa9afde	updated test because the new tokenizer removes URLs ... Former-commit-id: 35f472fcf9	2015-06-18 11:38:28 -04:00
Rob Speer	5b4107bd1d	tests for new wordfreq with full coverage ... Former-commit-id: df863a5169	2015-05-21 20:34:17 -04:00