iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 09:21:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

forgot about Italian

Browse Source 
Former-commit-id: 8f6cd0e57b
This commit is contained in:
Robyn Speer 2015-11-30 18:18:24 -05:00
parent ab8c2e2331
commit 064ee22a33
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
wordfreq_builder/wordfreq_builder/tokenizers.py
View File

@ -48,8 +48,8 @@ def cld2_surface_tokenizer(text):
# Low-frequency languages tend to be detected incorrectly. Keep a limited
# list of languages we're allowed to use here.
KEEP_THESE_LANGUAGES = {
'ar', 'de', 'el', 'en', 'es', 'fr', 'hr', 'id', 'ja', 'ko', 'ms', 'nl',
'pl', 'pt', 'ro', 'ru', 'sv', 'th'
'ar', 'de', 'el', 'en', 'es', 'fr', 'hr', 'id', 'it', 'ja', 'ko', 'ms',
'nl', 'pl', 'pt', 'ro', 'ru', 'sv', 'th'
}