Algorithm Tokenize ( string s ): 1. list ( string ) tokens := [ ] 2. while not i
ID: 3804106 • Letter: A
Question
Algorithm Tokenize(string s):
1.list(string) tokens := [ ]
2. while not isEmpty(s)
a. If s begins with a token, remove the longest possible token from the beginning of s and push that token onto the back of tokens
b. If head(s) is a whitespace character, pop(s).
3. return tokens
For example, below is a trace of the call tokenize("hello, world 3+1")
# it
tokens
s
0
[]
"hello, world 3+1"
1
["hello"]
", world 3+1"
2
["hello",","]
" world 3+1"
3
["hello",","]
"world 3+1"
4
["hello",",","world"]
" 3+1"
5
["hello",",","world"]
"3+1"
6
["hello",",","world","3"]
"+1"
7
["hello",",","world","3","+"]
"1"
8
["hello",",","world","3","+","1"]
""
Homework:
Implement Tokenize in Python. That is, write a Python function called tokenize such that if s is a tring for which Tokenize(s) is defined, then Tokenize(s) == tokenize(s).
Test cases:
Note the following
tokenize("hello world")
["hello", "world"]
tokenize("hello'world'foo")
["hello", "'world'", "foo"]
["hello", "'world'", "foo"]
['hello', ''world'', 'foo']
tokenize("'hello\'world'")
["'hello\'world'"]
tokenize("'hello world'")
["'hello world'"]
tokenize("3.33(33..")
not part of our agreement
tokenize("\")
["\"]
tokenize(" ")
[ ]
tokenize("'a'e'c'")
["'a'", "e", "'c'"]
tokenize("3.3+1")
["3.3","+","1"]
tokenize("''")
["''"]
tokenize("<59>=6")
["<","59",">=","6"]
tokenize("+")
["+"]
tokenize("foo'4$!\\h\'32\t'88")
["foo","'4$!\\h\'32\t'"],
["88"]]
tokenize("'hello'wor'ld")
not part of our agreement
For example, you could enter into repl.it or IDLE,
>> tokenize("'hello\'world'")== ["'hello\'world'"]
>> True
Send your tokenizer as an attached .py file
# it
tokens
s
0
[]
"hello, world 3+1"
1
["hello"]
", world 3+1"
2
["hello",","]
" world 3+1"
3
["hello",","]
"world 3+1"
4
["hello",",","world"]
" 3+1"
5
["hello",",","world"]
"3+1"
6
["hello",",","world","3"]
"+1"
7
["hello",",","world","3","+"]
"1"
8
["hello",",","world","3","+","1"]
""
Explanation / Answer
code:
import os
import sys
import numpy as np
def Tokenize(str):
if str==" " or str=="":
return []
li =[]
for i in str.split():
li.append(i)
return li
print Tokenize("Hello world 3 + 1")
ouitput:
mitali@Infoweave:~/Desktop/chegg$ python ass1.py
['Hello', 'world', '3', '+', '1']
Related Questions
drjack9650@gmail.com
Navigate
Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.