iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 17:31:41 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix typo in docstring

Browse Source
This commit is contained in:
Rob Speer 2015-08-26 16:24:35 -04:00
parent c4a2594217
commit 34375958ef
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
wordfreq/tokens.py
View File

@ -9,7 +9,7 @@ TOKEN_RE = regex.compile(r"""
# When we see characters that are Han ideographs (\p{IsIdeo}) or hiragana
# \p{Script=Hiragana}, we allow a sequence of those characters to be glued
# together as a single token. Without this case, the standard rule (case 2)
# would make each characte a separate token. This would be the correct
# would make each character a separate token. This would be the correct
# behavior for word-wrapping, but a messy failure mode for NLP
# tokenization.
#