69 Commits

Author	SHA1	Message	Date
Rob Speer	02d8b32119	process leeds and opensubtitles	2015-05-07 17:07:33 -04:00
Rob Speer	7e238cf547	abstract how we define build rules a bit	2015-05-07 16:59:28 -04:00
Rob Speer	d2f9c60776	WIP on more build steps	2015-05-07 16:49:53 -04:00
Rob Speer	16928ed182	add rules to count wikipedia tokens	2015-05-05 15:21:24 -04:00
Rob Speer	bd579e2319	fix the 'count' ninja rule	2015-05-05 14:06:13 -04:00
Rob Speer	5787b6bb73	add and adjust some build steps ... - more build steps for Wikipedia - rename 'tokenize_twitter' to 'pretokenize_twitter' to indicate that the results are preliminary	2015-05-05 13:59:21 -04:00
Rob Speer	61b9440e3d	add wiki-parsing process	2015-05-04 13:25:01 -04:00
Rob Speer	34400de35a	not using wordfreq.cfg anymore	2015-04-30 16:25:42 -04:00
Rob Speer	5437bb4e85	WIP on new build system	2015-04-30 16:24:28 -04:00
Rob Speer	2a1b16b55c	use script codes for Chinese	2015-04-30 13:02:58 -04:00
Rob Speer	4dae2f8caf	define some ninja rules	2015-04-29 17:13:58 -04:00
Rob Speer	14e445a937	WIP on Ninja build automation	2015-04-29 15:59:06 -04:00
Rob Speer	815d393b74	move commands into cli/ directory	2015-04-29 15:22:04 -04:00
Rob Speer	70c9e99ee4	handle multi-word stems correctly	2015-04-29 13:45:53 -04:00
Rob Speer	60a7c4d1ec	revise the process of building Wikipedia counts	2015-03-30 18:09:07 -04:00
Rob Speer	be2b68b1de	enable wordlist balancing, surface form counting	2015-02-17 13:43:22 -05:00
Rob Speer	fcd6044c2d	add utility for combining wordlists	2015-02-11 11:45:10 -05:00
Rob Speer	d3374a9fe1	command-line entry points	2015-02-10 12:28:29 -05:00
Rob Speer	693c35476f	Initial commit	2015-02-04 20:19:36 -05:00