Can I use python ast module for this?

Question

I'd like to write a program that modify python programs in this way:

change

"some literal string %" % SOMETHING

to

functioncall("some literal string %") % SOMETHING

Thanks,

You might have better luck using the 2to3 tool, which is designed for source-to-source transformations. — Mark Dickinson, Jun 18 '10 at 13:52

score 2 · Answer 1 · answered Jun 18 '10 at 14:15

It may be simpler with tokenize -- adapting the example in the docs,

import cStringIO
import tokenize

class Lookahead(object):

  def __init__(self, s):
    self._t = tokenize.generate_tokens(cStringIO.StringIO(s).readline)
    self.lookahead = next(self._t, None)

  def __iter__(self):
    return self

  def next(self):
    result = self.lookahead
    if result is None: raise StopIteration
    self.lookahead = next(self._t, None)
    return result


def doit(s):
  toks = Lookahead(s)
  result = []
  for toktype, tokvalue, _, _, _ in toks:
    if toktype == tokenize.STRING:
      pk = toks.lookahead
      if pk is not None and pk[0] == tokenize.OP and pk[1] == '%':
        result.extend([
            (tokenize.NAME, 'functioncall'),
            (tokenize.OP, '('),
            (tokenize.STRING, repr(tokvalue)),
            (tokenize.OP, ')')
        ])
        continue
    result.append((toktype, tokvalue))
  return tokenize.untokenize(result)


print doit('"some literal string %" % SOMETHING')

This prints functioncall ('"some literal string %"')%SOMETHING. The spacing is pretty peculiar (it takes much more effort to get the spacing just right -- but that is even worse for reconstructing sources from a modified AST), but it's just fine if all you're going to do is import / run the resulting code (not so fine if you want to get nicely readable and editable code -- but that's a big enough problem that I'd suggest a separate Q;-).

There seems to be unwanted quotes as compared to what the OP was asking. I believe that replacing repr(tokvalue) by tokvalue gives the desired result. — André, Oct 13 '15 at 19:40

score 1 · Answer 2 · answered Jun 18 '10 at 12:59

1

You can solve this with having to write a program. Instead simply use the best editor ever made: Emacs. Worth learning if you haven't already. With it you can solve this by using its regex-replace capability. The only trouble is that I rarely use regex's so I always forget the details of the cryptic syntax and have to still look it up :P I'll try to figure it again for you. Here's a link to the Search & Replace Info for Emacs - scroll down for using regex's

answered Jun 18 '10 at 12:59

Khorkrak

3,911
2
27
37

1

I need a way to do it melodramatically, so I can include it in another program as a kind of preprocessor. – Juanjo Conti Jun 18 '10 at 13:03
[Melodrama](http://en.wikipedia.org/wiki/Melodrama)tically, huh? But if you want to do it code-wise, then just use python's `re` module to handle the regular expressions. – JAB Jun 18 '10 at 14:42
Melodramatically I typed? F*ing spell checker. I was programatically. – Juanjo Conti Jun 18 '10 at 15:24
I'd expect you could just use re.sub in python to find matching patterns and replace them with whatever you want. AST is deprecated in python 2.6 and is adios in 3.0 so... [Python compiler.ast doc](http://docs.python.org/library/compiler.html) – Khorkrak Jun 18 '10 at 17:00
Regexs can easily handle this problem. As an Emacs user, I feel obliged to add that Emacs is not the only way to exercise regexs "easily". "Programmatic" options also can include many *nix tools ('sed', et.al.) as well as languages such as Python, Perl, etc. – JS. Sep 12 '13 at 18:28

score 0 · Answer 3 · edited May 23 '17 at 12:06

0

Here is another SO question that may be useful.

I gather that the ast module doesn't have a facility for returning to source code, but Armin Ronacher has written a module codegen which implements a to_source function for doing just that for ast nodes.

I haven't tried to do this myself.

edited May 23 '17 at 12:06

Community

1
1

answered Jun 18 '10 at 14:21

Matt Anderson

19,311
11
41
57

score 0 · Answer 4 · answered Jun 18 '10 at 14:53

import re

pattern = r'(".+? %")(?= %)'
oldstr = '"some literal string %" % SOMETHING'

newstr = re.sub(pattern, r'functioncall(\1)', oldstr)

Try something like that. (Though with file I/O, of course.) I haven't worked with ast at all yet, so I don't really know if using that would be any easier for something like this, but it seems to me that if you're just doing a simple search-replace and not actually doing a lot of complex parsing then there's no need to use ast.

Can I use python ast module for this?

4 Answers4