Python programming; pleasee help!! We want to build a tokenizer for simple expre

Question

Python programming; pleasee help!!

We want to build a tokenizer for simple expressions such as xpr = "res = 3 + x_sum*11". Such expressions comprise only three tokens, as follows: (1) Integer literals: one or more digits e.g., 3 11; (2) Identifiers: strings starting with a letter or an underscore and followed by more letters, digits, or underscores e.g., res x_sum; (3) Operators: = + * . Leading or trailing whitespace characters should be skipped.

(b) The problem with the above is that re.findall returns a list of all lexemes but not their respective tokens (as defined by the grammar). Modify your regex pattern so that each matched lexeme is in its own (regex) group. The list returned should now be as follows:

[('', 'res', ''), ('', '', '='), ('3', '', ''), ('', '', '+'), ('', 'x_sum', ''), ('', '', '*'), ('11', '', '')]

Explanation / Answer

As per your problem requirement, I could suggest you to use tokenize.tokenize

Please find the required below program which gives you the required output.

Here only you have to modify token.name as per your requirement in python and execute that.You will get the output.

Related Questions

Navigate

• Browse (All)
• Browse P
• Subjects

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.