Would love some assistance/fix for this python code (regarding pandas data-frame) please

Question

I am running into this warning in my Jupyter notebook and I am relatively new to python and coding in general. Any help would be appreciated :)

Warning/Error:

FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

Code causing the error:


dataCollection = pd.DataFrame()
for file in list_of_files:
    procData = pd.read_table(file, header=None, sep='\s+')
    procData.columns = columns
    procData['subject_id'] = int(file[-5])
    dataCollection = dataCollection.append(procData, ignore_index=True)

dataCollection.reset_index(drop=True, inplace=True)
dataCollection.head()

The output I am expecting would be a dataframe table with column headings.

The warning tells me to use pandas.concat but it's unclear on how I can use that in the above code — Sou, Dec 17 '22 at 21:04
Even if `append` was not getting deprecated, using it repeatedly has a quadratic complexity. `concat` should be used **on all DataFrames at once**. — mozway, Dec 17 '22 at 21:44

score 0 · Accepted Answer · answered Dec 17 '22 at 21:14

The error is telling you to use pandas.concat in place of frame.append.

dataCollection = pd.DataFrame()
for file in list_of_files:
    procData = pd.read_table(file, header=None, sep='\s+')
    procData.columns = columns
    procData['subject_id'] = int(file[-5])
    dataCollection = pd.concat([dataCollection, procData], ignore_index=True)

Of course, this is an inefficient way of concatenating an arbitrary set of frames. Get all the frames in a list, then call concat once.

frames = []
for f in list_of_files:
    procData = pd.read_table(f, header=None, sep='\s+')
    procData.columns = columns
    procData['subject_id'] = int(file[-5])
    frames.append(procData)

dataCollection = pd.concat(frames, ignore_index=rue)

Would love some assistance/fix for this python code (regarding pandas data-frame) please

1 Answers1