iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 09:21:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

correct the simple_tokenize docstring

Browse Source
This commit is contained in:
Rob Speer 2015-08-26 13:54:50 -04:00
parent 01b6403ef4
commit f7babea352
1 changed files with 2 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
wordfreq/tokens.py
View File

@ -55,9 +55,8 @@ def simple_tokenize(text):
ideograms and hiragana) relatively untokenized, instead of splitting each
character into its own token.
- It excludes punctuation, many classes of symbols, and "extenders" with
nothing to extend, from being tokens, but it allows miscellaneous symbols
such as emoji.
- It outputs only the tokens that start with a word-like character, or
miscellaneous symbols such as emoji.
- It breaks on all spaces, even the "non-breaking" ones.
"""