17 Commits

Author	SHA1	Message	Date
Joshua Chin	631a5f1b71	removed mkdir -p for many cases ... Former-commit-id: 98a7a8093b	2015-07-17 14:45:22 -04:00
Rob Speer	4771c12814	remove wiki2tokens and tokenize_wikipedia ... These components are no longer necessary. Wikipedia output can and should be tokenized with the standard tokenizer, instead of the almost-equivalent one in the Nim code.	2015-06-30 15:28:01 -04:00
Rob Speer	9a2855394d	fix comment and whitespace involving tokenize_twitter	2015-06-30 15:18:37 -04:00
Rob Speer	f305679caf	Switch to a centibel scale, add a header to the data	2015-06-22 17:38:13 -04:00
Joshua Chin	da93bc89c2	removed intermediate twitter file rules	2015-06-16 17:28:09 -04:00
Rob Speer	536c15fbdb	give mecab a larger buffer	2015-05-26 19:34:46 -04:00
Rob Speer	ffd352f148	correct a Leeds bug; add some comments to rules.ninja	2015-05-26 18:08:04 -04:00
Rob Speer	50ff85ce19	add Google Books data for English	2015-05-11 18:44:28 -04:00
Rob Speer	d6cc90792f	Makefile should only be needed for bootstrapping Ninja	2015-05-08 12:39:31 -04:00
Rob Speer	abb0e059c8	a reasonably complete build process	2015-05-07 19:38:33 -04:00
Rob Speer	d2f9c60776	WIP on more build steps	2015-05-07 16:49:53 -04:00
Rob Speer	16928ed182	add rules to count wikipedia tokens	2015-05-05 15:21:24 -04:00
Rob Speer	bd579e2319	fix the 'count' ninja rule	2015-05-05 14:06:13 -04:00
Rob Speer	5787b6bb73	add and adjust some build steps ... - more build steps for Wikipedia - rename 'tokenize_twitter' to 'pretokenize_twitter' to indicate that the results are preliminary	2015-05-05 13:59:21 -04:00
Rob Speer	5437bb4e85	WIP on new build system	2015-04-30 16:24:28 -04:00
Rob Speer	4dae2f8caf	define some ninja rules	2015-04-29 17:13:58 -04:00
Rob Speer	14e445a937	WIP on Ninja build automation	2015-04-29 15:59:06 -04:00