2

I want to be able to open a dataframe in a Spyder (Python IDE) window and quickly scroll down to the last record of a relatively large pandas dataframe (134,890 records in table). It's frustrating because Spyder only loads a certain number of records from large tables, and scrolling down takes a while.

Does anyone know a shortcut or option to quickly get to the end of the dataframe, without writing any code like df.tail()?

The following image shows a dataframe with data extending into 2018, but it takes me quite a while to scroll to the end. Every time I scroll to what should be the bottom, Spyder loads a few more records from the table, and I start scrolling down again...

Spyder variable explorer

Spyder view dataframe window

Sean McCarthy
  • 4,838
  • 8
  • 39
  • 61
  • Okay, I'll bite: why can't you write ANY code to solve this problem? Can you sort the index descending and put the last/most recent entry at the top? – Evan Jan 31 '18 at 18:08
  • @Evan, I certainly could write some relatively simple code, but I open dataframes quite often and I want to see the most recent dates very quickly. I often open several related dataframes in separate windows, to compare them. – Sean McCarthy Jan 31 '18 at 18:21
  • 1
    `df.tail(n)` is about as simple as it gets. If all you need are dates, can you `groupby` date or index? – Evan Jan 31 '18 at 18:34
  • @Evan, I agree that's very simple, but I want to open the dataframes on separate monitors and compare them side by side. I don't just want to print text to the console. Tx – Sean McCarthy Jan 31 '18 at 18:37

2 Answers2

1

I think I figured it out. I must keep pulling down on the scroll bar and wiggle it left and right until Spyder has loaded all the records. The key is not to let go of the scroll bar until Spyder has finished loading all the records. If I let go, Spyder will stop loading new records.

Sean McCarthy
  • 4,838
  • 8
  • 39
  • 61
1

(Spyder maintainer here) There's a simpler and much easier way to inspect the end of a Dataframe: pressing with the mouse on the name of the column you want to sort and the Dataframe will be sorted in reverse order if Pandas can do it.

Note: Sorting by index will be available in Spyder 4, to be released by the end of 2018.

Carlos Cordoba
  • 33,273
  • 10
  • 95
  • 124
  • That doesn't seem to work for the datetime index, actually. I click on the index column and the box turns blue but nothing happens. If I click on one of the values columns, it does work, however. – Sean McCarthy Feb 01 '18 at 15:38
  • 1
    @Sean_Calgary, right, I forgot that sorting by index is working only in our master branch (from which Spyder 4 will be released). I updated the answer with that info. – Carlos Cordoba Feb 01 '18 at 15:51