Using Pandas in Python on datasets too large for excel

Question

I had a quick application question on using pandas in python to analyze large excel sheets.

For data that have millions of rows (beyond Excel's limit), how can we deal with analyzing them through pandas?

I know excel lets you load data from a text file and have your excel spreadsheet "create a connection" to the source file without having to load all the millions of rows directly. If we call this excel spreadsheet using pandas in python, will we be able to use our filter operations (and all the other table data analysis operations we've learned ) on all the millions of rows from the source file? Or will it just execute on only what shows up on the excel sheet (assuming we have selected the "create a connection" option to the source text file )?

Is there a more efficient way of using pandas with sas files directly?

Does this answer your question? [pandas: Is there a max size, max no. of columns, max rows?](https://stackoverflow.com/questions/15455722/pandas-is-there-a-max-size-max-no-of-columns-max-rows) — Serge de Gosson de Varennes, Dec 03 '20 at 08:13

score 0 · Answer 1 · answered Dec 03 '20 at 09:27

0

I think getting files via small chunks can make the process efficient. Please look at this link

answered Dec 03 '20 at 09:27

Anuj Suhag

71
6

Using Pandas in Python on datasets too large for excel

1 Answers1