Commit Graph

  • eb08c0a951 add docstrings to chinese_ and japanese_tokenize Robyn Speer 2015-10-27 13:23:56 -0400
  • 49b8ba4be9 add docstrings to chinese_ and japanese_tokenize staging-20151105 code-review-20151105 Rob Speer 2015-10-27 13:23:56 -0400
  • e1f7a1ccf3 add docstrings to chinese_ and japanese_tokenize Rob Speer 2015-10-27 13:23:56 -0400
  • f4d865c0be Merge pull request #28 from LuminosoInsight/chinese-external-wordlist Lance Nathan 2015-10-19 18:21:52 -0400
  • f47249064f Merge pull request #28 from LuminosoInsight/chinese-external-wordlist staging-20151023 code-review-20151023 Lance Nathan 2015-10-19 18:21:52 -0400
  • ca00dfa1d9 Merge pull request #28 from LuminosoInsight/chinese-external-wordlist Lance Nathan 2015-10-19 18:21:52 -0400
  • 5fedd71a66 Define globals in relevant places Robyn Speer 2015-10-19 18:15:54 -0400
  • 668a985969 Define globals in relevant places Rob Speer 2015-10-19 18:15:54 -0400
  • a6b6aa07e7 Define globals in relevant places #28 Rob Speer 2015-10-19 18:15:54 -0400
  • 91a81c1bde clarify the tokenize docstring Robyn Speer 2015-10-19 12:18:12 -0400
  • f255eb5bd8 clarify the tokenize docstring Rob Speer 2015-10-19 12:18:12 -0400
  • bfc17fea9f clarify the tokenize docstring Rob Speer 2015-10-19 12:18:12 -0400
  • c9693c9502 Merge branch 'master' into chinese-external-wordlist Robyn Speer 2015-09-28 14:34:59 -0400
  • 8fea2ca181 Merge branch 'master' into chinese-external-wordlist Rob Speer 2015-09-28 14:34:59 -0400
  • 1793c1bb2e Merge branch 'master' into chinese-external-wordlist Rob Speer 2015-09-28 14:34:59 -0400
  • 6d5ead0b47 Merge pull request #29 from LuminosoInsight/code-review-notes-20150925 Andrew Lin 2015-09-28 13:53:50 -0400
  • d8422852f4 Merge pull request #29 from LuminosoInsight/code-review-notes-20150925 staging-20151009 code-review-20151009 Andrew Lin 2015-09-28 13:53:50 -0400
  • 15d99be21b Merge pull request #29 from LuminosoInsight/code-review-notes-20150925 Andrew Lin 2015-09-28 13:53:50 -0400
  • f3f66508bd Fix documentation and clean up, based on Sep 25 code review Robyn Speer 2015-09-28 12:58:20 -0400
  • 3bd1fe2fe6 Fix documentation and clean up, based on Sep 25 code review Rob Speer 2015-09-28 12:58:20 -0400
  • 44b0c4f9ba Fix documentation and clean up, based on Sep 25 code review #29 Rob Speer 2015-09-28 12:58:20 -0400
  • 7494ae27a7 fix missing word in rules.ninja comment Robyn Speer 2015-09-24 17:56:06 -0400
  • 7435c8f57a fix missing word in rules.ninja comment Rob Speer 2015-09-24 17:56:06 -0400
  • 9b1c4d66cd fix missing word in rules.ninja comment Rob Speer 2015-09-24 17:56:06 -0400
  • 8e963dc312 describe optional dependencies better in the README Robyn Speer 2015-09-24 17:54:52 -0400
  • 7c596de98a describe optional dependencies better in the README Rob Speer 2015-09-24 17:54:52 -0400
  • b460eef444 describe optional dependencies better in the README Rob Speer 2015-09-24 17:54:52 -0400
  • 960dc437a2 update and clean up the tokenize() docstring Robyn Speer 2015-09-24 17:47:16 -0400
  • 28381d5a51 update and clean up the tokenize() docstring Rob Speer 2015-09-24 17:47:16 -0400
  • 24b16d8a5d update and clean up the tokenize() docstring Rob Speer 2015-09-24 17:47:16 -0400
  • 4a4534c466 test_chinese: fix typo in comment Robyn Speer 2015-09-24 13:41:11 -0400
  • f89ac5e400 test_chinese: fix typo in comment Rob Speer 2015-09-24 13:41:11 -0400
  • 2a84a926f5 test_chinese: fix typo in comment Rob Speer 2015-09-24 13:41:11 -0400
  • e15a231401 Merge branch 'master' into chinese-external-wordlist Robyn Speer 2015-09-24 13:40:08 -0400
  • faf66e9b08 Merge branch 'master' into chinese-external-wordlist Rob Speer 2015-09-24 13:40:08 -0400
  • cea2a61444 Merge branch 'master' into chinese-external-wordlist Rob Speer 2015-09-24 13:40:08 -0400
  • e27a75029d Revert "Remove the no-longer-existent .txt files from the MANIFEST." Andrew Lin 2015-09-24 13:31:34 -0400
  • c53bb06988 Revert "Remove the no-longer-existent .txt files from the MANIFEST." staging-20150925 code-review-20150925 Andrew Lin 2015-09-24 13:31:34 -0400
  • cd0797e1c8 Revert "Remove the no-longer-existent .txt files from the MANIFEST." Andrew Lin 2015-09-24 13:31:34 -0400
  • bb4653f16f Merge pull request #27 from LuminosoInsight/chinese-and-more Andrew Lin 2015-09-24 13:25:21 -0400
  • 566a62abd5 Merge pull request #27 from LuminosoInsight/chinese-and-more Andrew Lin 2015-09-24 13:25:21 -0400
  • 710eaabbe1 Merge pull request #27 from LuminosoInsight/chinese-and-more Andrew Lin 2015-09-24 13:25:21 -0400
  • e7d46fb104 Revert a small syntax change introduced by a circular series of changes. Andrew Lin 2015-09-24 13:24:11 -0400
  • ee6df56514 Revert a small syntax change introduced by a circular series of changes. Andrew Lin 2015-09-24 13:24:11 -0400
  • 09597b7cf3 Revert a small syntax change introduced by a circular series of changes. #27 Andrew Lin 2015-09-24 13:24:11 -0400
  • 4d00f17477 don't apply the inferred-space penalty to Japanese Robyn Speer 2015-09-24 12:49:45 -0400
  • 1b7117952b don't apply the inferred-space penalty to Japanese Rob Speer 2015-09-24 12:49:45 -0400
  • db5eda6051 don't apply the inferred-space penalty to Japanese Rob Speer 2015-09-24 12:49:45 -0400
  • 6b163e5772 Revert "Remove the no-longer-existent .txt files from the MANIFEST." Andrew Lin 2015-09-23 13:02:40 -0400
  • 4ccfcdc1bd Revert "Remove the no-longer-existent .txt files from the MANIFEST." Andrew Lin 2015-09-23 13:02:40 -0400
  • bb70bdba58 Revert "Remove the no-longer-existent .txt files from the MANIFEST." Andrew Lin 2015-09-23 13:02:40 -0400
  • d215f79ea3 describe the use of lang in read_values Robyn Speer 2015-09-22 17:22:38 -0400
  • 88deef24f6 describe the use of lang in read_values Rob Speer 2015-09-22 17:22:38 -0400
  • f224b8dbba describe the use of lang in read_values Rob Speer 2015-09-22 17:22:38 -0400
  • e6e29a1c03 Make the jieba_deps comment make sense Robyn Speer 2015-09-22 17:19:00 -0400
  • 7cb310b28e Make the jieba_deps comment make sense Rob Speer 2015-09-22 17:19:00 -0400
  • 7c12f2aca1 Make the jieba_deps comment make sense Rob Speer 2015-09-22 17:19:00 -0400
  • b4628abb38 actually, still delay loading the Jieba tokenizer Robyn Speer 2015-09-22 16:54:39 -0400
  • d68dd9f568 actually, still delay loading the Jieba tokenizer Rob Speer 2015-09-22 16:54:39 -0400
  • 48734d1a60 actually, still delay loading the Jieba tokenizer Rob Speer 2015-09-22 16:54:39 -0400
  • 13642d6a4d replace the literal 10 with the constant INFERRED_SPACE_FACTOR Robyn Speer 2015-09-22 16:46:07 -0400
  • 0e4daa8472 replace the literal 10 with the constant INFERRED_SPACE_FACTOR Rob Speer 2015-09-22 16:46:07 -0400
  • 7a3ea2bf79 replace the literal 10 with the constant INFERRED_SPACE_FACTOR Rob Speer 2015-09-22 16:46:07 -0400
  • 01f9c07c33 remove unnecessary delayed loads in wordfreq.chinese Robyn Speer 2015-09-22 16:42:13 -0400
  • 5929975338 remove unnecessary delayed loads in wordfreq.chinese Rob Speer 2015-09-22 16:42:13 -0400
  • 4a87890afd remove unnecessary delayed loads in wordfreq.chinese Rob Speer 2015-09-22 16:42:13 -0400
  • db30d09947 load the Chinese character mapping from a .msgpack.gz file Robyn Speer 2015-09-22 16:31:50 -0400
  • 42ccba4fa6 load the Chinese character mapping from a .msgpack.gz file Rob Speer 2015-09-22 16:31:50 -0400
  • 6cf4210187 load the Chinese character mapping from a .msgpack.gz file Rob Speer 2015-09-22 16:31:50 -0400
  • fe8a6b51e7 document what this file is for Robyn Speer 2015-09-22 15:31:27 -0400
  • e12a42f38a document what this file is for Rob Speer 2015-09-22 15:31:27 -0400
  • 06f8b29971 document what this file is for Rob Speer 2015-09-22 15:31:27 -0400
  • 6802a4f89d fix README conflict Robyn Speer 2015-09-22 14:23:55 -0400
  • 76c4a8975a fix README conflict Rob Speer 2015-09-22 14:23:55 -0400
  • 5b918e7bb0 fix README conflict Rob Speer 2015-09-22 14:23:55 -0400
  • 9a007b9948 refactor the tokenizer, add include_punctuation option Robyn Speer 2015-09-15 13:26:09 -0400
  • 963e0ff785 refactor the tokenizer, add include_punctuation option Rob Speer 2015-09-15 13:26:09 -0400
  • e8e6e0a231 refactor the tokenizer, add include_punctuation option Rob Speer 2015-09-15 13:26:09 -0400
  • 1adbb1aaf1 add external_wordlist option to tokenize Robyn Speer 2015-09-10 18:09:41 -0400
  • e3a79ab8c9 add external_wordlist option to tokenize Rob Speer 2015-09-10 18:09:41 -0400
  • 669bd16c13 add external_wordlist option to tokenize Rob Speer 2015-09-10 18:09:41 -0400
  • f2be213933 Merge branch 'greek-and-turkish' into chinese-and-more Robyn Speer 2015-09-10 15:27:33 -0400
  • 7f92557a58 Merge branch 'greek-and-turkish' into chinese-and-more Rob Speer 2015-09-10 15:27:33 -0400
  • 3cb3061e06 Merge branch 'greek-and-turkish' into chinese-and-more Rob Speer 2015-09-10 15:27:33 -0400
  • f0c7c3a02c Lower the frequency of phrases with inferred token boundaries Robyn Speer 2015-09-10 14:16:22 -0400
  • a13f459f88 Lower the frequency of phrases with inferred token boundaries Rob Speer 2015-09-10 14:16:22 -0400
  • 5c8c36f4e3 Lower the frequency of phrases with inferred token boundaries Rob Speer 2015-09-10 14:16:22 -0400
  • 66f1afe4d7 Merge pull request #26 from LuminosoInsight/greek-and-turkish Andrew Lin 2015-09-10 13:48:33 -0400
  • 800039f0f8 Merge pull request #26 from LuminosoInsight/greek-and-turkish staging-20150911 code-review-20150911 Andrew Lin 2015-09-10 13:48:33 -0400
  • acbb25e6f6 Merge pull request #26 from LuminosoInsight/greek-and-turkish Andrew Lin 2015-09-10 13:48:33 -0400
  • c5d5b0b1fe In ninja deps, remove 'startrow' as a variable Robyn Speer 2015-09-10 13:46:19 -0400
  • e3cc8eaea9 In ninja deps, remove 'startrow' as a variable Rob Speer 2015-09-10 13:46:19 -0400
  • a4f8d11427 In ninja deps, remove 'startrow' as a variable #26 Rob Speer 2015-09-10 13:46:19 -0400
  • acddc3ca05 fix spelling of Marc Robyn Speer 2015-09-09 13:35:02 -0400
  • 5701c1165d fix spelling of Marc Rob Speer 2015-09-09 13:35:02 -0400
  • 2277ad3116 fix spelling of Marc Rob Speer 2015-09-09 13:35:02 -0400
  • 872556f7bb fixes based on code review notes Robyn Speer 2015-09-09 13:10:18 -0400
  • 9c08442dc5 fixes based on code review notes Rob Speer 2015-09-09 13:10:18 -0400
  • 354555514f fixes based on code review notes Rob Speer 2015-09-09 13:10:18 -0400
  • 3dd70ed1c2 fix SUBTLEX citations Robyn Speer 2015-09-08 17:43:16 -0400