1

I have a file that I read from which has definitions of ctypes that are used in a separate project. I can read the file and obtain all the necessary information to create a ctype that I want in Python like the name, fields, bitfields, ctype base class (Structure, Union, Enum, etc), and pack.

I want to be able to create a ctype class from the information above. I also want these ctypes to be pickleable.

I currently have two solutions, both of which I feel like are hacks.

Solution 1

Generate a Python code object in an appropriate ctype format by hand or with the use of something like Jinja2 and then evaluate the python code object.

This solution has the downside of using eval. I always try to stay away from eval and I don't feel like this is a good place to use it.

Solution 2

Create a ctype dynamically in a function like so:

from ctypes import Structure

def create_ctype_class(name, base, fields, pack):
    class CtypesStruct(base):
        _fields_ = fields
        _pack_ = pack
    CtypesStruct.__name__ = name
    return CtypesStruct

ctype = create_ctype_class('ctype_struct_name', ctypes.Structure,
                           [('field1', ctypes.c_uint8)], 4)

This solution isn't so bad, but setting the name of the class is ugly and the type cannot be pickled.

Is there a better way of creating a dynamic ctype class?

Note: I am using Python 2.7

blitzvergnugen
  • 196
  • 8
Righteous Mullet
  • 174
  • 1
  • 13
  • You could use a generic function such as `create_type = lambda name, *bases, **attrs: type(name, bases, attrs)`. Then use `ctype = create_type('ctype', Structure, _fields_=[('field1', c_uint8)], _pack_=4)`. – Eryk Sun Feb 17 '15 at 00:45
  • For pickling the name that's set in the module has to agree with the class name. Similarly for an array type, subclass `Array` and set the `_type_` and `_length_`, which is unfortunately less convenient than using `ctype * length`. Also, ctypes default `__reduce__` won't pickle an instance of any pointer type such as `c_char_p`, or an aggregate that contains one or more pointers. – Eryk Sun Feb 17 '15 at 00:46
  • Wouldn't your first comment still work if I had `_type_` and `_length_` given as attrs? I am in the process of developing a reduce method that works on these classes. – Righteous Mullet Feb 17 '15 at 01:00
  • Solution 2 worked great for me. One note for the noobs out there (like myself) `'create_ctype_class()` returns the class itself, not an instance of it. So you have to `my_ctype_class = create_ctype_class(...)` then `my_ctype = my_ctype_class()` to actually use it. – blitzvergnugen Jan 23 '18 at 16:02

2 Answers2

0

Solution 2 is probably your better option, though if you're also writing such classes statically, you may want to use a metaclass to deduplicate some of that code. If you need your objects to be pickleable, then you'll need a way to reconstruct them from pickleable objects. Once you've implemented such a mechanism, you can make the pickle module aware of it with a __reduce__() method.

Kevin
  • 28,963
  • 9
  • 62
  • 81
  • Ok. I guess I have to figure out how to setup the __reduce__() method for these classes then. I have been fiddling around with that for quite a while. – Righteous Mullet Feb 17 '15 at 00:44
  • [Here's a simple `__reduce__()` I wrote](https://bitbucket.org/NYKevin/nbtparse/src/354332a1ec78e0c7d37b5902b3fab6434459c90e/nbtparse/semantics/nbtobject.py?at=default#cl-262). The class already had a serialization mechanism, but it wasn't directly compatible with `__getstate__()`/`__setstate__()`, so I just hooked it up to `__reduce__()` instead. – Kevin Feb 17 '15 at 00:55
  • There's no need to override `__reduce__`, unless you need custom support for handling pointer types. Otherwise non-pointer data types already have a working `__reduce__` and `__setstate__`. Also, messing with metaclasses is more trouble than it's worth in this case because ctypes uses its own metaclasses that you'd have to subclass, e.g. `type(ctypes.Structure)`, `type(ctypes.Union)`, `type(ctypes.Array)`, `type(ctypes._SimpleCData)`, `type(ctypes._Pointer)`, and `type(ctypes._CFuncPtr)`. – Eryk Sun Feb 17 '15 at 02:29
  • @eryksun - I was not aware that ctypes were capable of being pickled. No ctype that I am parsing will have pointers in them. I still implemented `__reduce__()` strictly to get the dynamic property of these classes working similar to the answer http://stackoverflow.com/a/11493777/3221505 – Righteous Mullet Feb 17 '15 at 17:37
0

I would go with a variant of Solution 1. Instead of evaling code, create a directory with an __init__.py (i.e. a package), add it to your sys.path and write out an entire python module containing all of the classes. Then you can import them from a stable namespace which will make pickle happier.

You can either take the output and add it to your app's source code or dynamically recreate it and cache it on a target machine at runtime.

pywin32 uses an approach like this for caching classes generated from ActiveX interfaces.

David K. Hess
  • 16,632
  • 2
  • 49
  • 73
  • That won't unpickle cleanly unless you can guarantee the module will also be present, with exactly the same name and contents, on the unpickling machine (which may not be the same as the pickling machine). – Kevin Feb 17 '15 at 00:50
  • Sure, but the result could be added to source code control or you could dynamically recreate it on the target machine. – David K. Hess Feb 17 '15 at 00:53
  • That is a good solution to my question. I didn't mention this, but I have at least a thousand ctypes or more in an XML file that I am generating these ctypes from. This process will occur often so I don't want it to take more than 5-10 seconds and currently it takes about 5 or so to run it with a modified version of Solution 2. I don't think this solution would fulfill that requirement, but I didn't specify that in my original question. – Righteous Mullet Feb 17 '15 at 00:53
  • @David: I'm not aware of any hook `pickle` provides for such dynamic recreation. I suppose you could recreate during `__init__.py` or add an import hook, but IMHO those are both ugly. – Kevin Feb 17 '15 at 00:54
  • I wouldn't hook pickle - just set it up during app initialization before you ever get around to invoking pickle. – David K. Hess Feb 17 '15 at 01:05