How to insert a row to dataframe fast?

Question

I want to insert data to the first row of a dataframe and remove the last row from the dataframe. I need to do this operation a lot of times.

From these two questions. Q1Q2 I have some code like this:

targetDF = pd.concat([sourceDF[sourceDF.index == targetDate], targetDF[:]])
targetDF.drop(targetDF.tail(1).index, inplace = True)

This will do from:

                Volume     Today      Lag1      Lag2      Lag3      Lag4      Lag5  Direction
2000-05-22  23956801.0  0.338821  1.724165 -0.343514  3.191463  0.714161  0.000000        1.0
2000-05-19  50544773.0 -1.013452  0.338821  1.724165 -0.343514  3.191463  0.714161       -1.0
...                ...       ...       ...       ...       ...       ...       ...        ...
2000-01-04   3194413.0  7.397314  5.187307 -1.977310 -0.840434 -2.458993  1.666694        1.0

[100 rows x 8 columns]

To:

                Volume     Today      Lag1      Lag2      Lag3      Lag4      Lag5  Direction
2000-05-23  21822575.0  1.724165 -0.343514  3.191463  0.714161  0.000000 -1.754358        1.0
2000-05-22  23956801.0  0.338821  1.724165 -0.343514  3.191463  0.714161  0.000000        1.0
...                ...       ...       ...       ...       ...       ...       ...        ...
2000-01-05   6058531.0  5.187307 -1.977310 -0.840434 -2.458993  1.666694  1.408448        1.0

[100 rows x 8 columns]

From the answer of Question 1, pd.concat() is expansive. So I would like to ask if there is a better way to do that?

Thank you very much.

Answer 1

pandas.concat

Modification of what you were doing

targetDF = pd.concat([sourceDF.loc[[targetDate]], targetDF.iloc[:-1]])

---

pandas.DataFrame.append

Or use a DataFrame method

targetDF = sourceDF.loc[[targetDate]].append(targetDF.iloc[:-1])

Answer 2

Note that in Pandas the physical order of rows is not so important and may be changed using either sort_index or sort_values.

Note also that each row is identified by an index and just like in a database, it should be unique within the current DataFrame.

So my concept is:

• If you want to add a new row, add it with index value one bigger than the max index so far, using append (at the end).
• If you want to delete a row, delete the row with the lowest value of the index (from the beginning).

So the order of rows is opposite to your wish, but you still are able to process them in "your" order (from the just inserted row to the "oldest" row.

Just access rows in reverse order than the index. Something like:

df.loc[::-1]