0

I have a 5GB Dataframe (with thousands of columns), every time a user makes an input in my web app, I load and grab a column from this Dataframe and return some calculations.

My frontend is a Vue app and the backend is in Flask.

The Dataframe is a large matrix, so it exceeds the column limit of a database (at least the ones I try).

Where I could store this Dataframe to be able to load it in a fast way?

Filipe
  • 199
  • 1
  • 14

1 Answers1

0

Where did the data come from to populate the dataframe?

If it was a database for example, you could just grab the data the user needed at that time rather than putting all your data into memory.

On the other hand, if your data didn't come from a database then that would be my first suggestion. Put the data into a database, these things were built for that purpose. You can create a dataframe from an sql query (pandas.read_sql).

If you really must have 5g of data in memory, then perhaps an in memory database would be suitable. For this purpose Redis comes to mind.

You could either pickle the dataframe and store the whole thing as an object, or you could break it down to records and store the individually and pull down only the data the user requests. This would remove the requirement for your application to store 5g of data in memory all the time.

Grizzle
  • 519
  • 3
  • 13
  • The Dataframe is in a matrix shape, so it has thousands of columns and exceeds the column limit of a SQL database (at least the ones I try). I will grab the column that matches the user input, so it is important to maintain the whole dataset available. – Filipe Jan 09 '21 at 22:06
  • 1
    There are ways to deal with that, but that's more of a data modeling exercise. Break down your data into more manageable chucks (perhaps related columns are turned into their own table). The fact is storing all of your data in memory does not scale, 5gb isn't too bad by todays standard but what happens when it grows to 20GB or even 200GB. The infrastructure cost to keep the data in memory will quickly balloon out. – Grizzle Jan 09 '21 at 22:15