How to write a small tokenizer in Python?

Question

Normally, Python calls functions by

func(arg0, arg1)

But I would like to change to

func arg0 arg1

For example,

#Something...
cmd = input()
interpret(cmd)
#Something...

If I input 'func arg0 arg1', then I expect Python to execute func(arg0, arg1).

Args will contain string, so that we can't simply split words.

Actually, I would like to write some scripts to use on my mobile. So it would be a little annoying to type parentheses.

So you want to rewrite the Python parser to create your own language...? I think that's a bit out of scope here. — deceze, Jul 28 '16 at 09:11
@deceze No. Because I would like to use it on a terminal on my phone. It's annoying to type parentheses on my phone. — Zhipeng YANG, Jul 28 '16 at 09:13
So you just want to split the first word from the input, look up the function by that name, and call it with the rest of the arguments like `f(*args)`...? — deceze, Jul 28 '16 at 09:14
Yes. But arg may be a string like `'Hello, World!'. So I think it should not simply split them. Should have a little parser I suppose. — Zhipeng YANG, Jul 28 '16 at 09:15
@ZhipengYANG So it's just `()` that's bugging you - what about all the other symbols `'{}[]+-*/"#` etc... that you're almost always going to have to use? — Jon Clements, Jul 28 '16 at 09:17
F# sometimes allows you to omit parentheses in that situation. Have you considered trying F#? — TigerhawkT3, Jul 28 '16 at 09:21
@TigerhawkT3 No. Because on Android's Termux, it doesn't support neither F# nor Haskell. — Zhipeng YANG, Jul 28 '16 at 09:21
@deceze It's a little bit different to that question. I don't want to write an entirre language. I just want to simplify how I call functions on mobile. — Zhipeng YANG, Jul 28 '16 at 09:23
In the end you *are* looking to create a new mini language. If you need to take strings with spaces etc. into account, you need a parser, and *boom*, you're in DSL territory. — deceze, Jul 28 '16 at 09:24
@deceze As there are answers, how could I delete the question? — Zhipeng YANG, Jul 28 '16 at 09:30

janbrohl · Answer 1 · 2016-07-28T09:38:04.340

0

if none of the args contains whitespace you could do

fn_args=cmd.split()
python_code="%s(%s)" % (fn[0], ", ".join(fn_args[1:]))
eval(python_code)

Edit:

If it is not that simple you should have a look at https://docs.python.org/3/library/cmd.html and https://docs.python.org/3/library/argparse.html but these require some preparation before you can execute arbitrary code

Edit2:

If you do not need your args to be exact python, you could parse them as json with the standard library

you could do it like

import json
cmd='fname "a" "b" 1'
fn,sep,args=cmd.strip().partition(" ")
end=0
largs=[]
d=json.JSONDecoder()
while end < len(args):
    args=args[end:].strip()
    arg,end=d.raw_decode(args)
    largs.append(arg)
exec(fn)(*largs) # better look into exec docs

edited Jul 28 '16 at 09:38

answered Jul 28 '16 at 09:15

janbrohl

2,626
1
17
15

Actually there will be spaces and commas. – Zhipeng YANG Jul 28 '16 at 09:16
@ZhipengYANG Added links for not-that-simple cases – janbrohl Jul 28 '16 at 09:22
@ZhipengYANG Added JSON hack – janbrohl Jul 28 '16 at 09:38

score 0 · Answer 2 · answered Jul 28 '16 at 09:16

You can do this :

class  tryClass:
    def callFunction(self, arg, arg2):
        print("In call")
        print(arg)
        print(arg2)

input = str(input())
input = input.split(" ")
funcName = input[0]
my_cls = tryClass()

method = getattr(my_cls, funcName)
method(input[1], input[2])

If I put in input callFunction hello world it works :)

Zhipeng YANG · Answer 3 · 2016-07-29T12:27:32.247

All I want is a simple tokenizer. And I would like to run functions by calling eval(). So that's what I did for my project.

Here's the result:

>>> tokenizer('func 123 abc')
[('func', 'func'), ('arg', '123'), ('arg', 'abc')]
>>> tokenizer('func 123.5 abc')
[('func', 'func'), ('arg', '123.5'), ('arg', 'abc')]
>>> tokenizer('func 123.5 abc "Hello, World!"')
[('func', 'func'), ('arg', '123.5'), ('arg', 'abc'), ('arg', 'Hello, World!')]
>>> tokenizer("func 123.5 abc 'Hello, World!'")
[('func', 'func'), ('arg', '123.5'), ('arg', 'abc'), ('arg', 'Hello, World!')]

Attentsion: This may not suitable for everyone, this's not a complete parser or tokenizer.

Code:

def isNumber(cmd):
    try:
        int(cmd)
        return True
    except ValueError:
        try:
            float(cmd)
            return True
        except ValueError:
            return False
    return False

def isWord(cmd):
    if len(cmd) == 0:
        return False
    if cmd[0].isalpha():
        for i in cmd[1:]:
            if not i.isalpha() and i != '_' and i != '-':
                return False
        return True
    return False
def spaceParser(cmd):
    i = 0
    for i in range(len(cmd)):
        if cmd[i] == ' ':
            continue
        break
    return cmd[i:]

def funcNameParser(cmd):
    cmd = spaceParser(cmd)
    i = 0
    word = ''
    for i in range(len(cmd)):
        if cmd[i] != ' ':
            word += cmd[i]
        else:
            break
    if i + 1 > len(word):
        return (word, cmd[i:])
    return (word, cmd[i+1:])

def argumentParser(cmd):
    cmd = spaceParser(cmd)
    if cmd[0] == '\'':
        word = ''
        i = 0
        for i in range(1, len(cmd)):
            if cmd[i] != '\'':
                word += cmd[i]
            else:
                return (word, cmd[i+1:])
        assert False, 'Fatal exception: String not finished.'
    if cmd[0] == '"':
        word = ''
        i = 0
        for i in range(1, len(cmd)):
            if cmd[i] != '"':
                word += cmd[i]
            else:
                return (word, cmd[i+1:])
        assert False, 'Fatal exception: String not finished.'            
    i = 0
    word = ''
    for i in range(len(cmd)):
        if cmd[i] != ' ':
            word += cmd[i]
        else:
            break
    assert isWord(word) or isNumber(word), 'Fatal exception: Not a valid name.'
    if i + 1 > len(word):
        return (word, cmd[i:]) 
    return (word, cmd[i+1:])

def tokenizer(cmd):
    token = []
    result = funcNameParser(cmd)
    token += [('func', result[0])]
    while len(result[1]) != 0:
        result = argumentParser(result[1])
        token += [('arg', result[0])]
    return token

score 0 · Answer 4 · answered Jul 28 '16 at 17:16

The builtin shlex module is probably what you want:

>>> import shlex
>>> cmd = "func arg0 arg1 'arg2 has spaces'"
>>> list(shlex.shlex(cmd))
['func', 'arg0', 'arg1', "'arg2 has spaces'"]

If you can trust the input, then actually calling this will look like:

>>> tokens = list(shlex.shlex(cmd))
>>> # here is a stupid func function that reverses its input args
>>> func = lambda *args: print(*reversed(args))
>>> eval(tokens[0])(*tokens[1:])
'arg2 has spaces' arg1 arg0

How to write a small tokenizer in Python?

4 Answers4