0

I want to view a pandas dataframe in Spyder's Variable Explorer.

When I click on the dataframe 'Name' in the Variable Explorer window, a series of files are executed (spyder_kernel.py, datapub.py, datapub.py, and serialize.py), which are part of the anaconda package. However, while executing serialize.py, the following message shows up:

serialize_object

buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))

OverflowError: cannot serialize a bytes object larger than 4 GiB

The dataframe is also not displayed. Any solution to this? Thanks.

Carlos Cordoba
  • 33,273
  • 10
  • 95
  • 124
SKhan
  • 21
  • 3

1 Answers1

0

(Spyder maintainer here) The problem you mentioned was fixed in Python 3.4 (according to this answer), which added a new Pickle protocol.

However, this is a bit tricky to solve for us because Spyder and the kernel that powers its IPython console run in different processes and both use Python. We'll address this in Spyder 4 (to be released this year), by detecting if both are running in Python 3.4+ to be able to use the right Pickle protocol.

Carlos Cordoba
  • 33,273
  • 10
  • 95
  • 124
  • I'm using python 3.6. So to rephrase your answer - although the underlying problem was fixed, the fix has not been implemented in Spyder? Is there a work-around? How can I view a dataframe or its column attributes, besides the bland output in the ipython console? Thanks. – SKhan Jan 11 '19 at 15:37
  • It's not that the fix is not implemented in Spyder, it's that we can't do that to maintain compatibility with Python 2. The reason is a user can select as a Python interpreter for our IPython consoles a Python 2 version, but have Spyder running in Python 3, and viceversa. So we need to use a pickle protocol that works in both Python versions and that's not protocol 4, but 2. – Carlos Cordoba Jan 11 '19 at 15:48
  • However, on a second thought, we can let the console know what Python version Spyder is running on, so we can instruct it to use the right pickle protocol depending on its and Spyder's Python versions. But this work requires some important changes to the way we handle communications between the console and Spyder, so we can only address it in Spyder 4 because Spyder 3 is maintenance mode right now. I opened [this issue](https://github.com/spyder-ide/spyder-kernels/issues/73) in case you want to track it. – Carlos Cordoba Jan 11 '19 at 16:01