64 Commits

Author	SHA1	Message	Date
Rob Speer	5787b6bb73	add and adjust some build steps ... - more build steps for Wikipedia - rename 'tokenize_twitter' to 'pretokenize_twitter' to indicate that the results are preliminary	2015-05-05 13:59:21 -04:00
Rob Speer	61b9440e3d	add wiki-parsing process	2015-05-04 13:25:01 -04:00
Rob Speer	34400de35a	not using wordfreq.cfg anymore	2015-04-30 16:25:42 -04:00
Rob Speer	5437bb4e85	WIP on new build system	2015-04-30 16:24:28 -04:00
Rob Speer	2a1b16b55c	use script codes for Chinese	2015-04-30 13:02:58 -04:00
Rob Speer	4dae2f8caf	define some ninja rules	2015-04-29 17:13:58 -04:00
Rob Speer	14e445a937	WIP on Ninja build automation	2015-04-29 15:59:06 -04:00
Rob Speer	815d393b74	move commands into cli/ directory	2015-04-29 15:22:04 -04:00
Rob Speer	70c9e99ee4	handle multi-word stems correctly	2015-04-29 13:45:53 -04:00
Rob Speer	60a7c4d1ec	revise the process of building Wikipedia counts	2015-03-30 18:09:07 -04:00
Rob Speer	be2b68b1de	enable wordlist balancing, surface form counting	2015-02-17 13:43:22 -05:00
Rob Speer	fcd6044c2d	add utility for combining wordlists	2015-02-11 11:45:10 -05:00
Rob Speer	d3374a9fe1	command-line entry points	2015-02-10 12:28:29 -05:00
Rob Speer	693c35476f	Initial commit	2015-02-04 20:19:36 -05:00