Generating Python code with Hy macros

Question

I am trying to generate some python code from Hy. How is that done better?

I have tried several approaches. One is with a macro:

(defmacro make-vars [data]
  (setv res '())
  (for [element data]
    (setv varname (HySymbol (+ "var" (str element))))
    (setv res (cons `(setv ~varname 0) res)))
  `(do ~@res))

Then after capturing macroexpansion, I print python disassembly of a code.

However, it seems that with macros I am unable to pass variables, so that:

(setv vnames [1 2 3])
(make-vars vnames)

defines varv, varn, vara and so on, in stead of var1, var2, var3. It seems that correct invocation could be made with:

(macroexpand `(make-vars ~vnames))

but that seems to be excessively complex.

Other issue I have encountered is the necessity of HySymbol, which came as a big surprise. But I have really been hurt by that, when I tried second approach, where I made a function that returns quoted forms:

(defn make-faction-detaches [faction metadata unit-types]
  (let [meta-base (get metadata "Base")
        meta-pattern (get metadata "Sections")
        class-cand []
        class-def '()
        class-grouping (dict)]
    (for [(, sec-name sec-flag) (.iteritems meta-pattern)]
      ;; if section flag is set but no unit types with the section are found, break and return nothing
      (print "checking" sec-name)
      (if-not (or (not sec-flag) (any (genexpr (in sec-name (. ut roles)) [ut unit-types])))
              (break)
              ;; save unit types for section 
              (do
               (print "match for section" sec-name)
               (setv sec-grouping (list-comp ut [ut unit-types]
                                             (in sec-name (. ut roles))))
               (print (len sec-grouping) "types found for section" sec-name)
               (when sec-grouping
                 (assoc class-grouping sec-name sec-grouping))))
      ;; in case we finished the cycle
      (else
       (do
        (def
          class-name (.format "{}_{}" (. meta-base __name__) (fix-faction-string faction))
          army-id (.format "{}_{}" (. meta-base army_id) (fix-faction-string faction))
          army-name (.format "{} ({})" (fix-faction-name faction) (. meta-base army_name)))
         (print "Class name is" class-name)
         (print "Army id is" army-id)
         (print "Army name is" army-name)
         (setv class-cand [(HySymbol class-name)])
         (setv class-def [`(defclass ~(HySymbol class-name) [~(HySymbol (. meta-base __name__))]
                            [army_name ~(HyString army-name)
                             faction ~(HyString faction)
                             army_id ~(HyString army-id)]
                             (defn --init-- [self]
                               (.--init-- (super) ~(HyDict (interleave (genexpr (HyString k) [k class-grouping])
                                                                       (cycle [(HyInteger 1)]))))
                               ~@(map (fn [key]
                                        `(.add-classes (. self ~(HySymbol key))
                                                       ~(HyList (genexpr (HySymbol (. ut __name__))
                                                                         [ut (get class-grouping key)]))))
                                      class-grouping)))]))))
    (, class-def class-cand)))

That function takes metadata that looks like this in python:

metadata = [
    {'Base': DetachPatrol,
     'Sections': {'hq': True, 'elite': False,
                  'troops': True, 'fast': False,
                  'heavy': False, 'fliers': False,
                  'transports': False}}]

And takes a list of classes that have form of:

class SomeSection(object):
    roles = ['hq']

It required extensive usage of internal classes of hy, and I failed to properly represent True and False, resorting to HyInteger(1) and HyInteger(0) instead.

To get python code from this function, I run its result through disassemble.

To summarise:

1. What would be the best way to generate python code from Hy?
2. What is internal representation for True and False?
3. Can one call a function that processes its parameters and returns a quoted Hy form from a macro and how?

Answer 1

In Hy you generally don't need to generate Python code, since Hy is much better at generating Hy code, and it is just as executable. This is done all the time in Hy macros.

In the unusual case that you need to generate real Python and not just Hy, the best way is with strings, the same way you'd do it in Python. Hy compiles to Python's AST, not to Python itself. The disassembler is really just for debugging purposes. It doesn't always generate valid Python:

=> (setv +!@$ 42)
=> +!@$
42
=> (disassemble '(setv +!@$ 42) True)
'+!@$ = 42'
=> (exec (disassemble '(setv +!@$ 42) True))
Traceback (most recent call last):
  File "/home/gilch/repos/hy/hy/importer.py", line 193, in hy_eval
    return eval(ast_compile(expr, "<eval>", "eval"), namespace)
  File "<eval>", line 1, in <module>
  File "<string>", line 1
    +!@$ = 42
     ^
SyntaxError: invalid syntax
=> (exec "spam = 42; print(spam)")
42

The variable name +!@$ is just as legal as spam is in the AST, but Python's exec chokes on it because it is not a valid Python identifier.

If you understand and are okay with this limitation, you can use disassemble, but without macros. Ordinary runtime functions are allowed to take and generate (as you demostrated) Hy expressions. Macros are really just functions like this than run at compile time. It's not unusual in Hy for a macro to delegate part of its work to an ordinary function that takes a Hy expression as one of its arguments and returns a Hy expression.

The easiest way to create a Hy expression as data is to quote it with '. The backtick syntax for interpolating values is also valid even outside the body of a macro. You can use this in normal runtime functions too. But understand, you must insert quoted forms into the interpolation if you want to disassemble it, because that's what a macro would receives as arguments--the code itself, not its evaluated values. That's why you're using HySymbol and friends.

=> (setv class-name 'Foo)  ; N.B. 'Foo is quoted
=> (print (disassemble `(defclass ~class-name) True))
class Foo:
    pass

You can ask the REPL what types it uses for quoted forms.

=> (type 1)
<class 'int'>
=> (type '1)
<class 'hy.models.HyInteger'>
=> (type "foo!")
<class 'str'>
=> (type '"foo!")
<class 'hy.models.HyString'>
=> (type True)
<class 'bool'>
=> (type 'True)
<class 'hy.models.HySymbol'>

As you can see, True is just a symbol internally. Note that I was able to generate a HySymbol with just ', without using the HySymbol call. If your metadata file was written in Hy and made with quoted Hy forms in the first place, you wouldn't have to convert them. But there's no reason it has to be done at the last minute inside the backtick form. That could be done in advance by a helper function if that's what you'd prefer.

---

Followup

Can one call a function that processes its parameters and returns a quoted Hy form from a macro and how?

My original point was that a macro is the wrong tool for what you're trying to do. But to clarify, you can call a macro at runtime, by using macroexpand, as you already demonstrated. You can, of course, put the macroexpand call inside another function, but macroexpand must have a quoted form as its argument.

Also, the same question about dynamically generated dictionaries. Construction I have used looks horrible.

The dictionary part could be simplified to something more like

{~@(interleave (map HyString class-grouping) (repeat '1))}

While Python's dict is backed by a hash table, Hy's HyDict model is really just a list. This is because it doesn't represent the hash table itself, but the code that produces the dict. That's why you can splice into it just like a list.

However if possible, could you add an example of properly passing dynamically generated strings into the final quoted expression? As far as I understand, it can be done with adding one more assignment (that would add quotation), but is there a more elegant way?

Hy's models are considered part of the public API, they're just not used much outside of macros. It's fine to use them when you need to. Other Lisps don't make the same kind of distinction between code model objects and the data they produce. Hy does it this way for better Python interop. One could argue that the ~ syntax should do this conversion automatically for certain datatypes, but at present, it doesn't. [Update: On the current master branch, Hy's compiler will auto-wrap compatible values in a Hy model when it can, so you usually don't have to do this yourself anymore.]

HySymbol is appropriate for dynamically generating symbols from strings like you're trying to do. It's not the only way, but it's what you want in this case. The other way, gensym, is used more often in macros, but they can't be as pretty. You can call gensym with a string to give it a more meaningful name for debugging purposes, but it still has a numeric suffix to make it unique. You could, of course, assign HySymbol a shorter alias, or delegate that part to a helper function.

You can also convert it in advance, for example, the fragment

(def class-name (.format "{}_{}" (. meta-base __name__) ...

Could instead be

(def class-name (HySymbol (.format "{}_{}" (. meta-base __name__) ...

Then you don't have to do it twice.

(setv class-cand [class-name])
(setv class-def [`(defclass ~class-name ...

That probably makes the template easier to read.

---

Update

Hy master now mangles symbols to valid Python identifiers on compilation, so the hy2py tool and the astor disassembly should more reliably generate valid Python code, even if there are special characters in the symbols.