3

Is there a pure Python way of transposing a memoryview?

Python memoryviews can represent more than just a 1-dimensional chunk of bytes. They can represent multidimensional layouts, noncontiguous memory, complex element types, and more. For example, in the following code:

In [1]: import numpy

In [2]: x = numpy.array([[1, 2], [3, 4]])

In [3]: y = x.T

In [4]: a = memoryview(x)

In [5]: b = memoryview(y)

a and b are 2-by-2 multidimensional memoryviews:

In [6]: a.shape
Out[6]: (2, 2)

In [7]: b.shape
Out[7]: (2, 2)

and b represents a transpose of a, so a[i, j] and b[j, i] alias the same memory (which is cell i, j of the original x array):

In [8]: a[0, 1] = 5

In [9]: b[1, 0]
Out[9]: 5

In [10]: x
Out[10]: 
array([[1, 5],
       [3, 4]])

NumPy arrays support easy transposes, but NumPy arrays aren't the only sources of multidimensional memoryviews. For example, you can cast a single-dimensional memoryview:

In [11]: bytearr = bytearray([1, 2, 3, 4])

In [12]: mem = memoryview(bytearr).cast('b', (2, 2))

In [13]: mem.shape
Out[13]: (2, 2)

In [14]: mem[1, 0] = 5

In [15]: bytearr
Out[15]: bytearray(b'\x01\x02\x05\x04')

The memoryview format is flexible enough to represent a transpose of mem, like what b was to a in our earlier example, but there doesn't seem to be an easy transpose method in the memoryview API. Is there a pure-Python way of transposing arbitrary multidimensional memoryviews?

user2357112
  • 260,549
  • 28
  • 431
  • 505
jakirkham
  • 685
  • 5
  • 18
  • what do you mean by transpose? a `memoryview` represent a chunk of allocated memory in your hardware, which is not, and cannot be, a multidimensional array, which means it can't be transposed like a vector. – Mia Jun 03 '20 at 00:29
  • 2
    A `memoryview` of _what_? Why doesn't the "normal" way of transposing the thing — whatever it is — work? – martineau Jun 03 '20 at 01:20
  • 3
    That's not exactly true. A `memoryview` can have a shape (IOW be multidimensional) memory. This is how `memoryview`s of NumPy arrays work https://docs.python.org/3/library/stdtypes.html#memoryview.shape – jakirkham Jun 03 '20 at 18:54
  • I'm trying to figure out how to do this within the standard library (without dependencies). – jakirkham Jun 03 '20 at 18:55
  • 1
    @pkqxdd all true multidimensional arrays, e.g. `numpy` are implemented using a contiguous chunk of allocated memory... – juanpa.arrivillaga Jun 03 '20 at 22:55
  • 1
    @martineau: We shouldn't have to care what the underlying object is. The underlying object might not have a concept of transposes at all, or if it does, it might not support transposes without copying. Memoryview shape and stride metadata is flexible enough to represent transposes without copying, but the memoryview Python-level interface doesn't expose the functionality. – user2357112 Jun 03 '20 at 23:40
  • 1
    For example, it's possible to have a multidimensional memoryview backed by a `bytearray`, with something like `memoryview(bytearray(b'asdf')).cast('b', (2, 2))`. The bytearray is 1-dimensional, but it's still possible to transpose the memoryview. – user2357112 Jun 03 '20 at 23:46

2 Answers2

3

There's no good way without taking dependencies. With NumPy, it's fairly straightforward, as long as the memoryview doesn't have suboffsets:

transposed = memoryview(numpy.asarray(orig_memoryview).T)

orig_memoryview can be backed by anything - there doesn't have to be a NumPy array behind it.

Unlike the other answer, the resulting memoryview is backed by the same memory as the original memoryview. For example, with the following multidimensional memoryview:

In [1]: import numpy

In [2]: arr = numpy.array([[1, 2], [3, 4]])

In [3]: mem = memoryview(arr)

we can transpose it:

In [4]: transposed = memoryview(numpy.asarray(mem).T)

and writes to the transposed memoryview affect the original array:

In [5]: transposed[0, 1] = 5

In [6]: arr
Out[6]: 
array([[1, 2],
       [5, 4]])

Here, writing to cell 0, 1 of the transpose corresponds to cell 1, 0 of the original array.

This doesn't rely on the original memoryview being backed by a NumPy array. It works fine with memoryviews backed by other things, like bytearrays:

In [7]: x = bytearray([1, 2, 3, 4])

In [8]: y = memoryview(x).cast('b', (2, 2))

In [9]: transposed = memoryview(numpy.asarray(y).T)

In [10]: transposed[0, 1] = 5

In [11]: y[1, 0]
Out[11]: 5

In [12]: x
Out[12]: bytearray(b'\x01\x02\x05\x04')

Without NumPy or a similar dependency, I don't see a good way. The closest thing to a good way would be to use ctypes, but you'd need to hardcode the Py_buffer struct layout for that, and the exact layout of a Py_buffer struct is undocumented. (The field order and types don't quite match the order in which the fields are documented, or the types they're documented with.) Also, for a PIL-style array with suboffsets, there's no way to transpose the memoryview without copying data.

On the bright side, most cases where you'd be dealing with multidimensional memoryviews, you'd already have the dependencies you'd need to transpose them.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • Thanks for this answer! FWIW it appears there is a Python bug report discussing the possibility of transposing `memoryview`s or more accurately handling F-order `memoryview`s correctly. https://bugs.python.org/issue35845 – jakirkham Jun 05 '20 at 16:36
1

This might help you:

>>> import numpy as np
>>> import array
>>> a = array.array('l', [-11111111, 22222222, -33333333, 44444444])
>>> m = memoryview(a)
>>> m_copy =  np.array(m)[np.newaxis]
>>> m_copy
array([[-11111111,  22222222, -33333333,  44444444]])
>>> m_copy.T
array([[-11111111],
       [ 22222222],
       [-33333333],
       [ 44444444]])

Without using numpy:

import array
a = array.array('l', [-11111111, 22222222, -33333333, 44444444])
print(a)

#output:
array('l', [-11111111, 22222222, -33333333, 44444444])

m = memoryview(a)
a = [[x for x in m]]
result = list(map(list, zip(*a)))
print(result)

#output:
[[-11111111], [22222222], [-33333333], [44444444]]
Sabito stands with Ukraine
  • 4,271
  • 8
  • 34
  • 56