iskm/wordfreq
1
0
Fork 0
mirror of https://github.com/rspeer/wordfreq.git synced 2024-12-23 17:31:41 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

updated comments

Browse Source 
Former-commit-id: 131b916c57
This commit is contained in:
Joshua Chin 2015-07-17 14:50:12 -04:00
parent 4c7910246e
commit e8fa25cb73
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
tests/test.py
View File

@ -94,7 +94,7 @@ def test_failed_cB_conversion():
def test_tokenization(): def test_tokenization():
# We preserve apostrophes within words, so "can't" is a single word in the # We preserve apostrophes within words, so "can't" is a single word in the
# data, while the fake word "plan't" can't be found. # data
eq_(tokenize("can't", 'en'), ["can't"]) eq_(tokenize("can't", 'en'), ["can't"])
eq_(tokenize('😂test', 'en'), ['😂', 'test']) eq_(tokenize('😂test', 'en'), ['😂', 'test'])