7

I need to write a class that implements 32-bit unsigned integers the same way they work in C programming language. What I care about most are the binary shifts, but I generally want my class to:

  1. Have the same interface int has and works with int properly
  2. Any operation with my U32 class (int + U32, U32 + int etc) also return U32
  3. Be pure-python - I don't want to use NumPy, ctypes, etc.

As can be found in this answer, I got a solution that works under Python 2. Recently I tried to run it under Python 3 and noticed that while the following test code works fine under older versions of Python, Python 3 raises an error:

class U32:
    """Emulates 32-bit unsigned int known from C programming language."""

    def __init__(self, num=0, base=None):
        """Creates the U32 object.

        Args:
            num: the integer/string to use as the initial state
            base: the base of the integer use if the num given was a string
        """
        if base is None:
            self.int_ = int(num) % 2**32
        else:
            self.int_ = int(num, base) % 2**32

    def __coerce__(self, ignored):
        return None

    def __str__(self):
        return "<U32 instance at 0x%x, int=%d>" % (id(self), self.int_)

    def __getattr__(self, attribute_name):
        print("getattr called, attribute_name=%s" % attribute_name)
        # you might want to take a look here:
        # https://stackoverflow.com/q/19611001/1091116
        r = getattr(self.int_, attribute_name)
        if callable(r):  # return a wrapper if integer's function was requested
            def f(*args, **kwargs):
                if args and isinstance(args[0], U32):
                    args = (args[0].int_, ) + args[1:]
                ret = r(*args, **kwargs)
                if ret is NotImplemented:
                    return ret
                if attribute_name in ['__str__', '__repr__', '__index__']:
                    return ret
                ret %= 2**32
                return U32(ret)
            return f
        return r

print(U32(4) / 2)
print(4 / U32(2))
print(U32(4) / U32(2))

And here's the error:

Traceback (most recent call last):
  File "u32.py", line 41, in <module>
    print(U32(4) / 2)
TypeError: unsupported operand type(s) for /: 'U32' and 'int'

It looks like the getattr trick doesn't get called at all in Python 3. Why is that? How can I get this code working both under Python 2 and 3?

Community
  • 1
  • 1
d33tah
  • 10,999
  • 13
  • 68
  • 158

2 Answers2

10

Your Python 2 solution relied on old style class behaviour. Your Python 2 code would fail in the same manner as Python 3 were you to make your class inherit from object:

class U32(object):

This is because special methods are looked up on the type, not the object itself, for new-style classes. This behaviour change fixed several corner cases with the old model.

In practice this means that methods like __div__ are looked up directly on U32 itself, not as attributes on instances of U32, and the __getattr__ hook is not consulted.

Unfortunately, special method lookups also bypass any __getattr__ or __getattribute__ hooks. See the documentation on Special Method lookups:

In addition to bypassing any instance attributes in the interest of correctness, implicit special method lookup generally also bypasses the __getattribute__() method even of the object’s metaclass:

[...]

Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).

Your only option then, is to set all special methods dynamically on your class. A class decorator would do fine here:

def _build_delegate(name, attr, cls, type_):
    def f(*args, **kwargs):
        args = tuple(a if not isinstance(a, cls) else a.int_ for a in args)
        ret = attr(*args, **kwargs)
        if not isinstance(ret, type_) or name == '__hash__':
            return ret
        return cls(ret)
    return f

def delegated_special_methods(type_):
    def decorator(cls):
        for name, value in vars(type_).items():
            if (name[:2], name[-2:]) != ('__', '__') or not callable(value):
                continue
            if hasattr(cls, name) and not name in ('__repr__', '__hash__'):
                continue
            setattr(cls, name, _build_delegate(name, value, cls, type_))
        return cls
    return decorator

@delegated_special_methods(int)
class U32(object):
    def __init__(self, num=0, base=None):
        """Creates the U32 object.

        Args:
            num: the integer/string to use as the initial state
            base: the base of the integer use if the num given was a string
        """
        if base is None:
            self.int_ = int(num) % 2**32
        else:
            self.int_ = int(num, base) % 2**32
    def __coerce__(self, ignored):
        return None
    def __str__(self):
        return "<U32 instance at 0x%x, int=%d>" % (id(self), self.int_)

I updated the proxy function to handle multiple arguments correctly, and to auto-coerce back to your custom class if int is returned.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • After that, the answer will be complete and I'll be happy to accept it. – d33tah Dec 06 '13 at 18:28
  • @d33tah: I was getting to it. :-) I'll write you a specific one, in the meantime, take a look at this [older answer of mine](http://stackoverflow.com/a/11281744). – Martijn Pieters Dec 06 '13 at 18:31
  • Thank you for the link and for your time. Looking forward to the specific one :) – d33tah Dec 06 '13 at 18:32
  • @d33tah: Actually, I forgot about the fact that `__special__` special method lookups also ignore `__getattribute__` and `__getattr__`. Updated to give you an alternative. – Martijn Pieters Dec 06 '13 at 18:59
  • @d33tah: However, if *both* your arguments are `U32()` then the code still fails, as `int.__truediv__(U32(..))` will fail. Your hook method will need to take that into account. – Martijn Pieters Dec 06 '13 at 19:01
  • Strange... does this mean that it's impossible to implement in Python 3? I just tried your code and got an error: `TypeError: unsupported operand type(s) for <<: 'U32' and 'U32'`, so it's not exactly what I wanted. :/ – d33tah Dec 06 '13 at 19:08
  • @d33tah: No, it just means the previous answer wasn't very complete. Updated to work correctly for multiple arguments, and simplified the return handling too. – Martijn Pieters Dec 06 '13 at 19:14
  • Works great, thank you! Though, in the end, I figured that I could as well define a global M = 2 ** 32 and do all operations % M... – d33tah Dec 06 '13 at 19:23
0

Inherit from int and replace all operators you want to use:

(tested with Python 3.7)

class U32(int):
    MAXVALUE = 0xffffffff

    def __new__(cls, value):
        return int.__new__(cls, value & cls.MAXVALUE)

    def __add__(self, *args, **kwargs):
        return self.__new__(type(self),int.__add__(self, *args, **kwargs))

    def __radd__(self, *args, **kwargs):
        return self.__new__(type(self),int.__radd__(self, *args, **kwargs))

    def __sub__(self, *args, **kwargs):
        return self.__new__(type(self), int.__sub__(self, *args, **kwargs))

    def __rsub__(self,*args, **kwargs):
        return self.__new__(type(self),int.__rsub__(self, *args, **kwargs))

    def __mul__(self, *args, **kwargs):
        return self.__new__(type(self),int.__mul__(self, *args, **kwargs))

    def __rmul__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rmul__(self, *args, **kwargs))

    def __div__(self, *args, **kwargs):
        return self.__new__(type(self),int.__floordiv__(self, *args, **kwargs))

    def __rdiv__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rfloordiv__(self, *args, **kwargs))

    def __truediv__(self, *args, **kwargs):
        return self.__new__(type(self),int.__floordiv__(self, *args, **kwargs))

    def __rtruediv__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rfloordiv__(self, *args, **kwargs))

    def __pow__(self, *args, **kwargs):
        return self.__new__(type(self),int.__pow__(self, *args, **kwargs))

    def __rpow__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rpow__(self, *args, **kwargs))

    def __lshift__(self, *args, **kwargs):
        return self.__new__(type(self),int.__lshift__(self, *args, **kwargs))

    def __rlshift__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rlshift__(self, *args, **kwargs))

    def __rshift__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rshift__(self, *args, **kwargs))

    def __rrshift__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rrshift__(self, *args, **kwargs))

    def __and__(self, *args, **kwargs):
        return self.__new__(type(self),int.__and__(self, *args, **kwargs))

    def __rand__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rand__(self, *args, **kwargs))

    def __or__(self, *args, **kwargs):
        return self.__new__(type(self),int.__ror__(self, *args, **kwargs))

    def __ror__(self, *args, **kwargs):
        return self.__new__(type(self),int.__ror__(self, *args, **kwargs))

    def __xor__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rxor__(self, *args, **kwargs))

    def __rxor__(self, *args, **kwargs):
        return self.__new__(type(self),int.__rxor__(self, *args, **kwargs))

Now it's easy to inherit other uint types:

class U16(U32):
    def __new__(cls, value):
        cls.MAXVALUE = 0xFFFF
        return int.__new__(cls, value&cls.MAXVALUE)


class U8(U32):
    def __new__(cls, value):
        cls.MAXVALUE = 0xFF
        return int.__new__(cls, value&cls.MAXVALUE)

Outputs

type( U8(0) + 1 ) = <class '__main__.U8'>
U8(0xcde) = 0xde
U8(2) + 0xff = 0x1
U8(0) + 0xfff = 0xff
U8(0) - 2 = 0xfe
U8(0xf) * 32 = 0xe0
U8(0x7)**3 = 0x57
U8(8) / 3 = 0x2
U8(0xff)>>4 = 0xf
U8(0xff)<<4 = 0xf0
type( 1 + U8(0) ) = <class '__main__.U8'>

Example printer to get the output examples

def exampleprinter(vec):
    for v in vec:
        result = eval(v)
        if issubclass(type(result), int):
            result = hex(result)
        print(v,'=' , result)

examples = [
            # results are of type U8
            'type( U8(0) + 1 )',
            # Correct tranform on over / underflow
            'U8(0xcde)',
            'U8(2) + 0xff',
            'U8(0) + 0xfff',
            'U8(0) - 2',
            'U8(0xf) * 32',
            'U8(0x7)**3',  # 7 ** 3 = 0x157 -> 0x157 & 0xff = 0x57
            # division will floor any remainder
            'U8(8) / 3',
            # Shifts
            'U8(0xff)>>4',
            'U8(0xff)<<4',
            # when swap operations are defined (the 'r...' ones) returns also U8 on reverse use
            'type( 1 + U8(0) )',
            ]

exampleprinter(examples)

I created these to take over some calculations of existing cryptogrphic-algorithm that relied heavily on bit-shifts on uint32.

A noticeable option is the reflective (or swapping) operator functions. These are the ones with the r-prefix, like __radd__, __rsub__, etc. They come into action when our uint type is the second operator, like in 1 + U8(0xff). If __radd__ is implemented the result will be 0x0 and of type U8, if is not implemented the result is 0x100 and of type int. While U8(0xff)+1 = 0 remains the same (since __add__ is defined). For my use-case it was best to define the reflective functions also to ensure all results were of type U32.

Marco
  • 580
  • 7
  • 10