0

Sorry If I didn'T find the post that has the answer to this.

I have to dataframes and I want to drop rows that aren't in BOTH dataframes, columns to compare are datetimes, like this:

2012-03-05 17:00
2012-03-05 17:30
2012-03-05 18:00
2012-03-05 18:30

If a given time isnt in one of the dataframes I would like to drop them.

How to? (I found merge only capabel of dropping if one has them, I may miss something)

Edit: sorry for the lack of information. I use pandas and numpy, but read the dataframes from csv with pd (pandas), I dont want to merge them just drop out any rows that arent in BOTH of them. Goal is to teach neural network with given dataframes of financial timeseries movement, if I lack data from a given date somewhere, I just want to drop that, to make the two dataframe identical in size. I Edit in some rows in a min

Edit2: Here are the outs of top 5 rows from both df

   Unnamed: 0       close                date        high         low  \
0           0  602.898376 2016-09-23 17:00:00  605.300000  602.586208   
1           1  602.309326 2016-09-23 17:30:00  604.200000  602.289105   
2           2  602.138367 2016-09-23 18:00:00  603.899999  596.148975   
3           3  600.599976 2016-09-23 18:30:00  603.807928  600.600005   
4           4  603.527527 2016-09-23 19:00:00  603.807928  601.123776   

  Local_Date_Local_Time  Open_Price  High_Price  Low_Price  Close_Price  
0   2016-09-23 17:00:00      1.1216      1.1219     1.1202       1.1215   
1   2016-09-23 17:30:00      1.1216      1.1228     1.1213       1.1222   
2   2016-09-23 18:00:00      1.1222      1.1226     1.1216       1.1220   
3   2016-09-23 18:30:00      1.1221      1.1231     1.1216       1.1227   
4   2016-09-23 19:00:00      1.1227      1.1233     1.1225       1.1230

as I mentioned, I would like to drop columns where date and "Local_Date_Local_Time" can't be find in both (sorry for the name :D , it had Date and time in diff column and I just merged them, will replace with something a bit shorter probably)

example: 2017.01.01 15:00 isnt in df1, then it should be dropped from df2

Basic Human Being
  • 93
  • 10
  • A little more information about the problem would be helpful. What exactly are the dataframes? – squgeim May 12 '17 at 13:47
  • What module are you using for dataframes? Panda? Numpy? etc...? – MooingRawr May 12 '17 at 13:47
  • very similar to this: http://stackoverflow.com/questions/28901683/pandas-get-rows-which-are-not-in-other-dataframe so you want one df , why not just merge: `df1.merge(df2, how='inner')` – EdChum May 12 '17 at 13:47
  • I dont want one df, I want two with same date rows – Basic Human Being May 12 '17 at 14:01
  • I test neural networks on financial data, df data type of column I wish to compare: datetime64[ns] ,I hope it helps, if you miss something from the edit and edit2 pls ask – Basic Human Being May 12 '17 at 14:02
  • http://stackoverflow.com/questions/30010529/python-pandas-remove-all-elements-of-one-dataframe-that-arent-included-in-anot question has been asked... – Basic Human Being May 12 '17 at 16:46

0 Answers0