Algorithm Tokenize ( string s ): 1. list ( string ) tokens := [ ] 2. while not i
ID: 3835469 • Letter: A
Question
Algorithm Tokenize(string s):
1. list(string) tokens := [ ]
2. while not isEmpty(s)
a. If s begins with a token, remove the longest possible token from the beginning of s and push that token onto the back of tokens
b. If head(s) is a whitespace character, pop(s).
3. return tokens
Homework:
Implement Tokenize in Python. That is, write a Python function called tokenize such that if s is a string for which Tokenize(s) is defined, then Tokenize(s) == tokenize(s).
Test cases:
tokenize("hello world")
["hello", "world"]
tokenize("hello'world'foo")
["hello", "'world'", "foo"]
["hello", "'world'", "foo"]
['hello', ''world'', 'foo']
tokenize("'hello\'world'")
["'hello\'world'"]
tokenize("'hello world'")
["'hello world'"]
tokenize("3.33(33..")
not part of our agreement
tokenize("\")
["\"]
tokenize(" ")
[ ]
tokenize("'a'e'c'")
["'a'", "e", "'c'"]
tokenize("3.3+1")
["3.3","+","1"]
tokenize("''")
["''"]
tokenize("<59>=6")
["<","59",">=","6"]
tokenize("+")
["+"]
tokenize("foo'4$!\\h\'32\t'88")
["foo","'4$!\\h\'32\t'"],
["88"]]
tokenize("'hello'wor'ld")
not part of our agreement
For example, you could enter into repl.it or IDLE,
>> tokenize("'hello\'world'")== ["'hello\'world'"]
>> True
tokenize("hello world")
["hello", "world"]
tokenize("hello'world'foo")
["hello", "'world'", "foo"]
["hello", "'world'", "foo"]
['hello', ''world'', 'foo']
tokenize("'hello\'world'")
["'hello\'world'"]
tokenize("'hello world'")
["'hello world'"]
tokenize("3.33(33..")
not part of our agreement
tokenize("\")
["\"]
tokenize(" ")
[ ]
tokenize("'a'e'c'")
["'a'", "e", "'c'"]
tokenize("3.3+1")
["3.3","+","1"]
tokenize("''")
["''"]
tokenize("<59>=6")
["<","59",">=","6"]
tokenize("+")
["+"]
tokenize("foo'4$!\\h\'32\t'88")
["foo","'4$!\\h\'32\t'"],
["88"]]
tokenize("'hello'wor'ld")
not part of our agreement
Explanation / Answer
$ python -m tokenize hello.py 0,0-0,0: ENCODING 'utf-8' 1,0-1,3: NAME 'def' 1,4-1,13: NAME 'say_hello' 1,13-1,14: OP '(' 1,14-1,15: OP ')' 1,15-1,16: OP ':' 1,16-1,17: NEWLINE ' ' 2,0-2,4: INDENT ' ' 2,4-2,9: NAME 'print' 2,9-2,10: OP '(' 2,10-2,25: STRING '"Hello, World!"' 2,25-2,26: OP ')' 2,26-2,27: NEWLINE ' ' 3,0-3,1: NL ' ' 4,0-4,0: DEDENT '' 4,0-4,9: NAME 'say_hello' 4,9-4,10: OP '(' 4,10-4,11: OP ')' 4,11-4,12: NEWLINE ' ' 5,0-5,0: ENDMARKER ''
Related Questions
drjack9650@gmail.com
Navigate
Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.