Can PANDAS be used to extract data from csv with file size more than 500MB. Are there alternate dataframe framework to work with large datasets

Asked Jun 05 '20 at 04:45

Active Jun 09 '20 at 05:58

Viewed 121 times

What are the PANDAS limitation to read large datasets in terms of size and number of records. Are there different libraries for large datasets

edited Jun 09 '20 at 05:58

asked Jun 05 '20 at 04:45

mtranpm

1

Does this answer your question? [Maximum size of pandas dataframe](https://stackoverflow.com/questions/23569771/maximum-size-of-pandas-dataframe) – rj-nirbhay Jun 05 '20 at 05:44
Welcome to stack overflow! Please [take the tour](https://stackoverflow.com/tour) and check out this guide on [how to ask](https://stackoverflow.com/help/how-to-ask) a question on the site. In particular, try to be more specific with your question. How large? What type of data? What's the actual issue you're trying to solve? – Michael Delgado Jun 05 '20 at 05:55
Thanks Michael for your guidance. We come across log data from various monitoring system monitoring the applications and if we have to solve identifying anomalies, the historical data run into GB's. I am trying to setup the pipeline and wanted to know if pandas is the best framework to read the files or there are alternate framework specifically to work with large file for EDA. – mtranpm Jun 09 '20 at 06:04

0 Answers0