14

I would like to pack all the data in a list into a single buffer to send over a UDP socket. The list is relatively long, so indexing each element in the list is tedious. This is what I have so far:

NumElements = len(data)
buf = struct.pack('d'*NumElements,data[0],data[1],data[2],data[3],data[4])

But I would like to do something more pythonic that doesn't require I change the call if I added more elements to the list... something like:

NumElements = len(data)
buf = struct.pack('d'*NumElements,data)  # Returns error

Is there a good way of doing this??

user1636547
  • 304
  • 1
  • 3
  • 13

2 Answers2

16

Yes, you can use the *args calling syntax.

Instead of this:

buf = struct.pack('d'*NumElements,data)  # Returns error

… do this:

buf = struct.pack('d'*NumElements, *data) # Works

See Unpacking Argument Lists in the tutorial. (But really, read all of section 4.7, not just 4.7.4, or you won't know what "The reverse situation…" is referring to…) Briefly:

… when the arguments are already in a list or tuple but need to be unpacked for a function call requiring separate positional arguments… write the function call with the *-operator to unpack the arguments out of a list or tuple…

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • Ah that's just what I needed! Thank you! – user1636547 May 04 '13 at 00:26
  • 2
    `'d' * NumElements` isn't very good practice, it could end up making a very large string, only to have to waste time parsing, only to free afterwards. Instead include the number directly in the string: `buf = struct.pack("{0:d}d".format(NumElements), *data)` – ideasman42 Apr 28 '15 at 19:38
  • @ideasman42: Good point. But with a huge number of values, unpacking them into 500000 arguments might be as much of a problem as passing 'd'*500000, so you might want to test that against `b''.join(struct.pack('d', elem) for elem in data)`, or consider using something like `array` or `ctypes` instead of `struct`. – abarnert Apr 28 '15 at 21:16
  • @abarnert, of course if performance is a priority... its worth looking into alternatives as you suggest, even so, when an API is provided that exposes a convenient & fast way to handle data. I'm not sure why you'd use the slower option. The example you give using `b''.join` is the slowest method by far, See benchmark https://gist.github.com/ideasman42/662dd741f31eaf33b006 Also, since this is an answer to a question for someone learning the API, its worth using good-practices. – ideasman42 Apr 28 '15 at 22:21
  • @ideasman42: My point isn't that `b''.join(…)` is going to be the fastest, it's that you shouldn't try to guess the fastest in advance; you should look at all of the reasonable options (with the same algorithmic complexity) and test them to see which really _is_ fastest. Also, you really want to test with `timeit`, not a loop over `time.time`; read the docs on the module for why. – abarnert Apr 28 '15 at 22:58
  • @abarnert, Yep, agree re: optimization, however, when there is an option to allocate a string, potentially unknown amount of memory, parse every character... then throw it away. If theres a convenient alternative which avoids doing that... just use it, theres really no good reason not to. Python devs dont do this in their code *(not counting tests)*, and probably if you submitted code to Python.org standard library it probably wouldn't pass review. – ideasman42 Apr 28 '15 at 23:25
  • @ideasman42: Well, sure, but we don't need to improve the OP's code until it meets the quality of the stdlib, we just need to fix the problem he's asking about. Of course adding comments explaining other things that might be worth improving is definitely helpful (that's why I upvoted your comment), but within the answer there's a tough balance between failing to correct irrelevant but bad things and swamping the asker in things he didn't ask about. (I think I usually err on the latter side, but if you think I did the opposite here, I can edit the answer.) – abarnert Apr 29 '15 at 00:05
2

The format string for struct.pack(...) and struct.unpack(...) allows to pass number(representing count) in front of type, with meaning how many times is the specific type expected to be present in the serialised data:

Simple case

data = [1.2, 3.4, 5.6]
struct.pack('3d', data[0], data[1], data[2])
struct.pack('3d', *[1.2, 3.4, 5.6])

or more generally:

data = [1.0, 1.234, 1.9, 3.14, 6.002, 7.4, 9.2]
struct.pack('{}d'.format(len(data)), *data)
PeterB
  • 101
  • 3