iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 09:21:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

updated monolingual_tokenize_file docstring, and removed unused argument

Browse Source
This commit is contained in:
Joshua Chin 2015-06-18 10:20:54 -04:00
parent 34e9512517
commit 18b53f6071
1 changed files with 9 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
wordfreq_builder/wordfreq_builder/tokenizers.py
View File

@ -93,11 +93,16 @@ def fix_entities(text):
def monolingual_tokenize_file(in_filename, out_filename, language,
tokenizer, line_reader=last_tab,
token_filter=lowercase_text_filter,
sample_proportion=100):
sample_proportion=1):
"""
Apply a tokenizer that can distinguish different languages, but only
keep the lines that are in the language we're asking for.
Process a file by running it through the given tokenizer, only keeping
lines of the language we're asking for, and inserting newlines
to mark the token boundaries.
`line_reader` is applied to each line before it given to the tokenizer
Only the first line out of every `sample_proportion` lines are run through
then tokenizer.
"""
with open(in_filename, encoding='utf-8', errors='replace') as in_file:
with open(out_filename, 'w', encoding='utf-8') as out_file: