1

This answer explains how you would unpickle multiple items in a file.

However, I'm sending pickled objects across the network. If two pickled objects are sent in rapid succession, they could end up read into the same buffer.

How do I replicate the behavior in the linked answer with bytes objects? Is there a well-defined terminator I can split on, or a way to "advance" the bytes object?

Ex:

test = pickle.dumps("hello")
test += pickle.dumps("world")

print(pickle.loads(test)) # Prints "hello" -- how do I get "world"?
ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
tonysdg
  • 1,335
  • 11
  • 32
  • 1
    If you don't have anything in place to separate your pickles from each other, you probably don't have anything in place to make sure your pickles stay in one piece, either. Make sure you don't end up trying to load half a pickle, or half of two pickles put together. – user2357112 Mar 14 '18 at 21:22
  • @user2357112: I ultimately ended up fixing the length of messages I'm passing at 8192 bytes -- anything longer raises an exceptions, shorter gets padded. It's a hack, but it solves the problem for now (it's a research prototype, so it's an acceptable hack). That solves the "half a pickle" problem -- as for the "half of two pickle" problems, I avoid that by using a message queue for each network connection. – tonysdg Mar 15 '18 at 17:27

1 Answers1

2

Follow the pattern in your linked answer, but use io.BytesIO to make an in-memory file-like object, e.g.:

# Changed to receive open file-like object instead of name
def load(fileobj):
    while True:
        try:
            yield pickle.load(fileobj)
        except EOFError:
            break

test = pickle.dumps("hello")
test += pickle.dumps("world")

with io.BytesIO(test) as f:
    for obj in load(f):
        ... do stuff with obj ...
ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • @user2357112: If you don't break out of loop, it will load them all. That's the point of the infinite loop in the generator, and the point of the answer the OP linked. – ShadowRanger Mar 14 '18 at 21:08
  • I reverted the edit because I'm intentionally mimicking [the OP's linked answer](https://stackoverflow.com/a/28745948/5003848); I agree the name `load` isn't great, but I'd prefer to perform minimal changes to keep the concepts in sync. – ShadowRanger Mar 14 '18 at 21:11
  • I was actually in the middle of editing the linked answer too, to change the function name and add a bit more explanation. – user2357112 Mar 14 '18 at 21:20