iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 09:21:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix typo in docstring

Browse Source 
Former-commit-id: 34375958ef
This commit is contained in:
Robyn Speer 2015-08-26 16:24:35 -04:00
parent af29fc4f88
commit 0b5d2cdca9
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
wordfreq/tokens.py
View File

@ -9,7 +9,7 @@ TOKEN_RE = regex.compile(r"""
# When we see characters that are Han ideographs (\p{IsIdeo}) or hiragana
# \p{Script=Hiragana}, we allow a sequence of those characters to be glued
# together as a single token. Without this case, the standard rule (case 2)
# would make each characte a separate token. This would be the correct
# would make each character a separate token. This would be the correct
# behavior for word-wrapping, but a messy failure mode for NLP
# tokenization.
#