455 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Rob Speer	d991373c1d	Work on making Japanese tokenization use MeCab consistently ... Former-commit-id: 05cf94d1fd	2015-05-27 18:10:25 -04:00
Rob Speer	cbe3513e08	Tokenize Japanese consistently with MeCab	2015-05-27 17:44:58 -04:00
Rob Speer	536c15fbdb	give mecab a larger buffer	2015-05-26 19:34:46 -04:00
Rob Speer	5de81c7111	fix build rules for Japanese Wikipedia	2015-05-26 18:08:57 -04:00
Rob Speer	3d5b3d47e8	fix version in config.py	2015-05-26 18:08:46 -04:00
Rob Speer	ffd352f148	correct a Leeds bug; add some comments to rules.ninja	2015-05-26 18:08:04 -04:00
Rob Speer	e4e146f22f	Merge branch 'master' into newbuild ... Conflicts: setup.py wordfreq/build.py wordfreq/config.py Former-commit-id: 0e5156e162	2015-05-21 20:41:47 -04:00
Rob Speer	b807d01f8f	rebuild data ... Former-commit-id: 84e5edcea1	2015-05-21 20:36:15 -04:00
Rob Speer	a1c31d3390	remove old tests ... Former-commit-id: 410912d8f0	2015-05-21 20:36:09 -04:00
Rob Speer	24a8e5531b	allow more language matches; reorder some parameters ... Former-commit-id: b42594fa5f	2015-05-21 20:35:02 -04:00
Rob Speer	5b4107bd1d	tests for new wordfreq with full coverage ... Former-commit-id: df863a5169	2015-05-21 20:34:17 -04:00
Rob Speer	c953fc1626	update README, another setup fix ... Former-commit-id: dd41e61c57	2015-05-13 04:09:34 -04:00
Rob Speer	5cbc0d0f94	update dependencies ... Former-commit-id: f13cca4d81	2015-05-12 12:30:01 -04:00
Rob Speer	6f61cac4cb	restore missing line in setup.py ... Former-commit-id: bb18f741e2	2015-05-12 12:24:18 -04:00
Rob Speer	1c65cb9f14	add new data files from wordfreq_builder ... Former-commit-id: 35aec061de	2015-05-11 18:45:47 -04:00
Rob Speer	50ff85ce19	add Google Books data for English	2015-05-11 18:44:28 -04:00
Rob Speer	c707b32345	move some functions to the wordfreq package	2015-05-11 17:02:52 -04:00
Rob Speer	9cd6f7c5c5	WIP: burn stuff down ... Former-commit-id: 9b63e54471	2015-05-08 15:28:52 -04:00
Rob Speer	d0d777ed91	use a more general-purpose tokenizer, not 'retokenize'	2015-05-08 12:40:14 -04:00
Rob Speer	35128a94ca	build.ninja knows about its own dependencies	2015-05-08 12:40:06 -04:00
Rob Speer	d6cc90792f	Makefile should only be needed for bootstrapping Ninja	2015-05-08 12:39:31 -04:00
Rob Speer	b541fe68e1	Merge branch 'ninja-build' ... Conflicts: wordfreq_builder/cmd_count_twitter.py wordfreq_builder/cmd_count_wikipedia.py	2015-05-08 00:01:01 -04:00
Rob Speer	2f14417bcf	limit final builds to languages with >= 2 sources	2015-05-07 23:59:04 -04:00
Rob Speer	1b7a2b9d0b	fix dependency	2015-05-07 23:55:57 -04:00
Rob Speer	abb0e059c8	a reasonably complete build process	2015-05-07 19:38:33 -04:00
Rob Speer	02d8b32119	process leeds and opensubtitles	2015-05-07 17:07:33 -04:00
Rob Speer	7e238cf547	abstract how we define build rules a bit	2015-05-07 16:59:28 -04:00
Rob Speer	d2f9c60776	WIP on more build steps	2015-05-07 16:49:53 -04:00
Lance Nathan	1bde55d516	Tweak to previous variable name fix ... Former-commit-id: e8a1548d93	2015-05-06 17:57:10 -04:00
Lance Nathan	d577c9e9c9	Merge pull request #6 from LuminosoInsight/ftfy4 ... Clean data with ftfy v4 Former-commit-id: 4632ffb177	2015-05-06 17:32:45 -04:00
Lance Nathan	b82b183c7a	Merge pull request #5 from LuminosoInsight/dutch-201504 ... Better Dutch surface-form data Former-commit-id: 5f05b52fe5	2015-05-06 17:15:21 -04:00
Rob Speer	f000ac2f1d	fix reused variable name ... Former-commit-id: 506073030a	2015-05-06 17:06:37 -04:00
Rob Speer	16928ed182	add rules to count wikipedia tokens	2015-05-05 15:21:24 -04:00
Rob Speer	bd579e2319	fix the 'count' ninja rule	2015-05-05 14:06:13 -04:00
Rob Speer	5787b6bb73	add and adjust some build steps ... - more build steps for Wikipedia - rename 'tokenize_twitter' to 'pretokenize_twitter' to indicate that the results are preliminary	2015-05-05 13:59:21 -04:00
Rob Speer	c439d492a5	set version number to 0.8 ... Former-commit-id: 2f3bb955d1	2015-05-05 12:05:00 -04:00
Rob Speer	d7ea4c420c	Merge branch 'dutch-201504' into ftfy4 ... Conflicts: setup.py Former-commit-id: 24a7c73e6d	2015-05-05 12:04:44 -04:00
Rob Speer	0cc89b1afa	require ftfy 4 ... Former-commit-id: 70b2c678ea	2015-05-05 12:04:13 -04:00
Rob Speer	61b9440e3d	add wiki-parsing process	2015-05-04 13:25:01 -04:00
Rob Speer	34400de35a	not using wordfreq.cfg anymore	2015-04-30 16:25:42 -04:00
Rob Speer	5437bb4e85	WIP on new build system	2015-04-30 16:24:28 -04:00
Rob Speer	2a1b16b55c	use script codes for Chinese	2015-04-30 13:02:58 -04:00
Rob Speer	732c932ac7	v0.7: make a proper Dutch 'surfaces' list ... Former-commit-id: 873ace87db	2015-04-30 13:01:24 -04:00
Rob Speer	4dae2f8caf	define some ninja rules	2015-04-29 17:13:58 -04:00
Rob Speer	14e445a937	WIP on Ninja build automation	2015-04-29 15:59:06 -04:00
Rob Speer	815d393b74	move commands into cli/ directory	2015-04-29 15:22:04 -04:00
Rob Speer	5d14d24738	always use surface forms	2015-04-29 15:17:00 -04:00
Rob Speer	4c44872d15	Merge branch 'master' into dutch-201503 ... Conflicts: wordfreq/build.py Former-commit-id: 6cf46ee5aa	2015-04-29 14:36:24 -04:00
Rob Speer	70c9e99ee4	handle multi-word stems correctly	2015-04-29 13:45:53 -04:00
Rob Speer	d3c41fd8d8	start a new multilingual wordlist called 'stems' ... So far, this wordlist is only in Dutch. Former-commit-id: af5f65b328	2015-03-31 15:59:30 -04:00