Good alternative to Pandas .append() method, now that it is being deprecated?

Question

I use the following method a lot to append a single row to a dataframe. One thing I really like about it is that it allows you to append a simple dict object. For example:

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])

# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)

Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is:

# Create the new row as its own dataframe
df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] })
df = pd.concat([df, df_new_row])

So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. :( Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?

Answer 1

Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List's "append" method are very efficient and won't be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design - that is why they deprecated the method

Answer 2

I also like the append method. But you can do it in one line with a list of dicts

df = pd.concat([df, pd.DataFrame.from_records([{ 'a': 1, 'b': 2 }])])

or using loc and tuples for values on DataFrames with incremenal ascending indexes

df.loc[len(df), ['a','b']] = 1, 2

or maybe

df.loc[len(df), df.columns] = 3, 4

Answer 3

If you want to use concat instead:

append

outputxlsx = outputxlsx.append(df, ignore_index=True)

concat

outputxlsx = pd.concat([outputxlsx, df], ignore_index=True)

Answer 4

I was facing a similar issue. The other solutions weren't really working for me. I'm leaving this answer here as an additional possibility to deal with the issue since this is the first google result for certain searches and I myself ended here at least for the second time.

In my case the data is not a dict but just a list of values for a known set of parameters. I want to add the parameter values to a dataframe as rows because this way I can access a series of all the values for one parameter via df[parameter].

I start with an empty DataFrame:

parameters = ['a', 'b', 'c', 'd', 'e', 'f']
df = pd.DataFrame(columns=parameters)

df:

        a   b   c   d   e   f

With append I could add rows very convenient like so:

new_row = pd.Series([1,2,3,4,5,6], index=parameters, name='row1')
df.append(new_row)

df:

        a   b   c   d   e   f
row1    1   2   3   4   5   6

With pd.concat I found this to deliver the same result in very similar way:

new_row = pd.DataFrame([1,2,3,4,5,6], columns=['row1'], index=parameters).T
df = pd.concat((df, new_row))

The key was to create a (1,n) dataframe from the 1d data and then transpose it to match the other dataframe.

Answer 5

For those, like me, who want a descriptive function rather than lots of one-liners, here is an option based on @Rafael Gaitan above.

def appendDictToDF(df,dictToAppend):
  df = pd.concat([df, pd.DataFrame.from_records([dictToAppend])])
  return df

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])

# Appending a row
df= appendDictToDF(df,{ 'a': 1, 'b': 2 })

Answer 6

# Deprecated issue has been resolved

# Creating an empty dataframe
df = pd.DataFrame(columns=['a', 'b'])
print("df columns:", df)

# Appending a row
df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True)
print("df column Values :", df)

# Create the new row as its own dataframe
df_new_row = pd.DataFrame.from_records({ 'a': [3], 'b': [4] })
df = pd.concat([df, df_new_row])
print("pd concat with two df's :", df)

Answer 7

You can use the following command

#If your index is a string
df.loc["name of the index"] = pd.Series({"Column 1" : Value1, "Column 2" : Value2,
"Column 3" : Value3, "Column 4" : Value4, ...})

#If your index is a number
df.loc[len(df)] = pd.Series({"Column 1" : Value1, "Column 2" : Value2,
"Column 3" : Value3, "Column 4" : Value4, ...})

Just keep in mind that the changes will be stored in the initial dataframe.

Answer 8

If you want to use chained operations in pandas to append new rows, you could use something like this:

import pandas as pd

new_row = pd.DataFrame({"sex": "male", "age": 40, "survived": False, "name": "Alex"}, index=[0])

(df_dict
  .assign(name=['Alice', 'Bob', 'Charlie'])
  .drop("pclass", axis=1)
  .pipe(lambda df_: pd.concat([df_, new_row], ignore_index = True))
)

For your example, I would use a custom append_row function.

import pandas as pd

def append_row(df1, d):
    df2 = pd.DataFrame(d, index=[0])
    return pd.concat([df1, df2], ignore_index = True)

df = pd.DataFrame(columns=['a', 'b'])
(df
    .pipe(append_row, {'a': 1, 'b': 2 })
)