shelve raises EOFError after a few iterations of pickle/unpickle

Question

I'm using the shelve module to save some data. The values of the keys of my shelve object are set objects.

Pickling snippet:

import shelve

other_links = {
'Blue Estate The Game, 72': 'https://store.steampowered.com/account/ackgift/A5e5AB6C5050A331B?redeemer=esmcfofp%40skinsgifts.com', 
'Hope Lake, 75': 'https://store.steampowered.com/account/ackgift/70E2E6Ce9F1733265?redeemer=esmcfofp%40skinsgifts.com', 
'Forget Me Not: My Organic Garden, 74': 'https://store.steampowered.com/account/ackwift/A14373949D126B37?redeemer=esmcfofp%40skinsgifts.com', 
'Until I Have You, 74': 'https://store.steampowered.com/account/ackgift/69030w5ECB8D0F07F?redeemer=esmcfofp%40skinsgifts.com', 
'Lex Mortis, 74': 'https://store.steampowered.com/account/ackgift/2760w1D8B48EB3601?redeemer=esmcfofp%40skinsgifts.com', 
'I am Bread, 72': 'https://store.steampowered.com/account/ackgift/1CEC5e2D2BEF20C41?redeemer=esmcfofp%40skinsgifts.com', 
'Lumini, 72': 'https://store.steampowered.com/account/ackgift/472F108Aw0609C215?redeemer=esmcfofp%40skinsgifts.com'
}
links = shelve.open('links', writeback = True)
for key, value in other_links.items():
    db_value = links.get(key)
    if not db_value:
        db_value = set()
    db_value.add(value)
    links[key] = db_value
links.close()

after a few dumps the EOFError exception is raised:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: 'Blue Estate The Game, 72'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
  File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "draft.py", line 161, in get_links
    send_links(client)
  File "draft.py", line 363, in send_links
    pickle_links(links)
  File "draft.py", line 288, in pickle_links
    db_value = links.get(key)
  File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\shelve.py", line 106, in get
    return self[key]
  File "C:\Users\user\AppData\Local\Programs\Python\Python35\lib\shelve.py", line 114, in __getitem__
    value = Unpickler(f).load()
EOFError: Ran out of input

Here is what I got in my shelve database in binary represantation:

{b'Lumini, 72': b'\x80\x03cbuiltins\nset\nq\x00]q\x01(Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/885EB5D41BEE5C06?redeemer=enugkpbh%40skinsgifts.comq\x02Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/2E98304B098FB501?redeemer=dtwggqot%40skinsgifts.comq\x03Xb\x00\x00\x00htt', 
b'Until I Have You, 74': b'\x80\x03cbuiltins\nset\nq\x00]q\x01(Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/C891F7EDCBE76C7A?redeemer=zpuvbsbi%40skinsgifts.comq\x02Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/02BCC505A7540D62?redeemer=enugkpbh%40skinsgifts.comq\x03Xb\x00\x00\x00htt', 
b'Hope Lake, 75': b'\x80\x03cbuiltins\nset\nq\x00]q\x01(Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/ED73ABF9D1D20620?redeemer=zpuvbsbi%40skinsgifts.comq\x02Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/367AFE87AF5CFE51?redeemer=dtwggqot%40skinsgifts.comq\x03Xb\x00\x00\x00htt', 
b'I am Bread, 72': b'\x80\x03cbuiltins\nset\nq\x00]q\x01(Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/40B2AC2B9CC73520?redeemer=xvkvctwm%40skinsgifts.comq\x02Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/868A11AE5F5CAC39?redeemer=dtwggqot%40skinsgifts.comq\x03Xb\x00\x00\x00htt', 
b'Forget Me Not: My Organic Garden, 74': b'\x80\x03cbuiltins\nset\nq\x00]q\x01(Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/FDD9F850DDB70369?redeemer=xvkvctwm%40skinsgifts.comq\x02Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/A310E30EFA33FF40?redeemer=dtwggqot%40skinsgifts.comq\x03Xb\x00\x00\x00htt', 
b'Blue Estate The Game, 72': b'\x80\x03cbuiltins\nset\nq\x00]q\x01(Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/4F56053C47479926?redeemer=zpuvbsbi%40skinsgifts.comq\x02Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/BD574CB54CEFB36A?redeemer=xvkvctwm%40skinsgifts.comq\x03Xb\x00\x00\x00htt', 
b'Lex Mortis, 74': b'\x80\x03cbuiltins\nset\nq\x00]q\x01(Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/09CCA4C8D4E57036?redeemer=xvkvctwm%40skinsgifts.comq\x02Xb\x00\x00\x00https://store.steampowered.com/account/ackgift/EA6E67E98ECDE852?redeemer=zpuvbsbi%40skinsgifts.comq\x03Xb\x00\x00\x00htt'}

What I see is that the third element of every value of the database starts with 'htt' and the value itself doesn't have a binary ending.

Great question. Well written and contains all the relevant info to reproduce. Made my day. — Mad Physicist, Nov 11 '16 at 18:46
Actually, could you show code for constructing and pickling the set? — Mad Physicist, Nov 11 '16 at 18:47
It's still a good question. The missing information is not an obvious omission. It's more just to satisfy my curiosity about a thing. — Mad Physicist, Nov 11 '16 at 19:10
Can you include in your example the use of the `shelve` module? And is the value of `set_as_bytes` the result of calling `pickle.dumps`, or did you come by that value via some other means? — larsks, Nov 11 '16 at 19:15
The code demonstrating how you're generating the pickled store isn't self-contained -- could you try to make it a proper reproducer anyone can copy-and-paste to run? — Charles Duffy, Nov 11 '16 at 19:34
Using your code I am unable to reproduce the problem. The `other_links` dictionary is shelved successfully, and I can extract binary values from the database using `dbm.open(...)` and individually unpickle all of them. — larsks, Nov 11 '16 at 20:44
Yes,, but after a few dumps (2 and more other_links dictionary dumps) to the database in my script the shell file gets corrupted. Additional code of my script is not needed, because all of the dictionaries are sure to be like the ones in my question. — Berman, Nov 12 '16 at 04:18
WFM here. Your traceback mentions a thread. Could something else be messing with `links` or the `'links'` file as the same time? Like, multiple instances of that code running concurrently? — ivan_pozdeev, Nov 12 '16 at 06:29
Possible duplicate of [Python Pickling Dictionary EOFError](http://stackoverflow.com/questions/10263564/python-pickling-dictionary-eoferror) — ivan_pozdeev, Nov 12 '16 at 06:36
^^ If this is about multiple threads sharing the entities without any locking. — ivan_pozdeev, Nov 12 '16 at 06:37
Thank you for the reply. I've decided explicitly to track every dump. I will add more info if the error is raised again. — Berman, Nov 12 '16 at 11:15

score 0 · Answer 1 · answered Nov 16 '16 at 12:33

I haven't been encountering the error so far. I think the error could occur at the moments when I forgot to close the file in interactive shell. Nevertheless, I did some tests leaving the file open and fetch and write some data in it - the error wasn't raised.

shelve raises EOFError after a few iterations of pickle/unpickle

1 Answers1