0

I need to read in multiple large .csv's (20k rows x 6k columns) and store them in a dataframe.

This thread has excellent examples that have worked for me in the past with smaller files.

Such as:

pd.concat((pd.read_csv(f,index_col='Unnamed: 0') for f in file_list))

Other more direct approaches that I have attempted is:

frame = pd.DataFrame()
list_ = []
for file_ in file_list:
    print(file_)
    df = pd.read_csv(file_,index_col=0)
    list_.append(df)
df = pd.concat(list_)

However all the solutions revolve around creating a list of all the csv files as individual df's and then using pd.concat() at the end over all of the df's.

As far as I can tell it's this approach which is causing a memory error when concat'ing ~20 of these df's.

  • How could I get past this and perhaps append each df as I go?

Example of file_list:

/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_26.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_30.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_25.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_19.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_27.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_18.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_28.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_23.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_03.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_24.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_29.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_04.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_20.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_22.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_06.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_05.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_01.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_06_02.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_31.csv
/realtimedata/orderbooks/bitfinex/btcusd/bitfinex_btcusd_orderbook_2018_05_21.csv
swifty
  • 1,182
  • 1
  • 15
  • 36

1 Answers1

1

Your CSVs are still manageably sized, so I would assume the issue is with misaligned headers.

I'd recommend reading in your DataFrames without any header, so concatenation is aligned.

list_ = []
for file_ in file_list:
    df = pd.read_csv(file_, index_col=0, skiprows=1, header=None)
    list_.append(df)

df = pd.concat(list_)
cs95
  • 379,657
  • 97
  • 704
  • 746
  • Unfortunately it did work for ~80% of those files, but failed again concatenating the entire file list. Is it completely unreasonable for this to blow up memory? I will continue to debug and check each file. – swifty Jun 08 '18 at 06:47
  • Split them in half and read in half the files each into a separate df,, consumed 14GB on a 16GB machine and would not concat :( – swifty Jun 08 '18 at 08:10