1

I can't seem to figure this out - I have a dataframe that I'm trying to iterate on each row but when I iterate it is converting my dataframe into a series.

df = pd.DataFrame(np.random.rand(5, 5)*10, columns=list('ABCDE')).astype(int)


for index, row in df.iterrows():
  print(row['A'],row['B'] )
  
print("Row shape - ", row.shape)
print("Row type - ", type(row))
print("DF shape - ", df.shape)
print("DF type - ", type(df))

0 1
6 1
6 4
9 6
6 2
Row shape -  (5,)
Row type -  <class 'pandas.core.series.Series'>
DF shape -  (5, 5)
DF type -  <class 'pandas.core.frame.DataFrame'>

I tried to_frames but it didn't work:

print("row after to frame ", row.to_frame().shape)
row after to frame  (5, 1)

My problem is, I need to send it to a function one row at a time(therefore using iterrows()) and need it to retain the same structure as the dataframe I'm iterrows.

The problem when I send it to the other function(transform using an existing scaler), it's rejecting it because it's shape is different than the dataframes. So in my example, for each row I'd like to send it in a shape like this (1, 5) instead of (5,).

Is that possible?

Lostsoul
  • 25,013
  • 48
  • 144
  • 239
  • Usually it is not good idea to use `df.iterrows`, you can use `apply` method. Can you tell me what function you are passing two. Then, I can write an answer. – Grayrigel Oct 14 '20 at 15:26
  • @Grayrigel just passing to a sklearn scaler but it expects the same type of dataframe used to originally scale. So I fit_transformed it with the entire data. Then as I get new, I need to do quick transforms against the fitted scaler. – Lostsoul Oct 14 '20 at 15:28
  • @Grayrigel funny thing is the question* I used to figure out iterrows's second answer says not to do it. In my case I do need to process one row at a time via a function (because the current data depends on the previous rows result from the function) * https://stackoverflow.com/questions/16476924/how-to-iterate-over-rows-in-a-dataframe-in-pandas – Lostsoul Oct 14 '20 at 15:41
  • It would really nice if you can given an example of transformation you want to do after fit_transforming the data. – Grayrigel Oct 14 '20 at 15:45
  • @Grayrigel it was the same transformation of my original data. I just did a standardscaler wrapped in a column transformer. Nothing fancy. – Lostsoul Oct 14 '20 at 15:57

1 Answers1

1

you can use transpose, for example in your code:

print("row after to frame ", row.to_frame().T.shape)

The output will be in the shape of (1,5) as desired. But, as Grayrigel mention in a comment, the recommended way is to use the apply function.

Daniel Ben Zaken
  • 117
  • 1
  • 9