merging two datasets based on common id and date within interval range

Question

I have two datasets: DF1 - data frame which lists heads of states (leader_id) of countries (country_code) and an interval of their time in office (office_interval). DF2 - data frame where every observation is an event that has an ID (event_ID) country (country_code) and date it occurred (event_date)

Data:

library(lubridate)

#Leader DF
leader_id <- c("Adam","Bob","Charlie","Derek", "Edgar")
country_code <- c(1,1,2,2,3)
office_interval <- c(interval(ymd("1900-01-01"), ymd("1905-01-01")), 
                     interval(ymd("1910-01-01"), ymd("1915-01-01")),
                     interval(ymd("1920-01-01"), ymd("1925-01-01")),
                     interval(ymd("1930-01-01"), ymd("1935-01-01")),
                     interval(ymd("1940-01-01"), ymd("1945-01-01")))
DF1 <- data.frame(leader_id, country_code, office_interval)

#Event DF
event_id <- c(1,1,2,3,3)
country_code <- c(1,2,2,1,3)
event_date <- c(as.Date("1901-01-02"), 
                as.Date("1920-01-02"), 
                as.Date("1921-01-02"),
                as.Date("1911-01-02"),
                as.Date("1941-02-02"))
DF2 <- data.frame(event_id, country_code, event_date)

I would like to take create a new column in DF2 that takes the leaderid from DF1 based on each row in DF2 that occur within a leaders office_interval in the same country.

DF2 should look like this afterward:

  event_id country_code event_date leader_id
1        1            1 1901-01-02      Adam
2        1            2 1920-01-02   Charlie
3        2            2 1921-01-02   Charlie
4        3            1 1911-01-02       Bob
5        3            3 1941-02-02     Edgar

I've tried some solutions from here but I cannot get any of them to work.

score 2 · Answer 1 · answered Jan 20 '20 at 13:28

Here is a solution maybe can work for your purpose

idx <- sapply(1:nrow(DF2), function(k) which(DF2$event_date[k] %within% DF1$office_interval & DF2$country_code[k]%in% DF1$country_code))
DF2$leader_id <- DF1$leader_id[idx]

such that

> DF2
  event_id country_code event_date leader_id
1        1            1 1901-01-02      Adam
2        1            2 1920-01-02   Charlie
3        2            2 1921-01-02   Charlie
4        3            1 1911-01-02       Bob
5        3            3 1941-02-02     Edgar

interesting solution! – LeroyFromBerlin Jan 20 '20 at 13:30 — LeroyFromBerlin, Jan 20 '20 at 13:30

Ronak Shah · Accepted Answer · 2020-01-20T13:34:23.207

1

We can left_join DF2 and DF1 by "country_code" and keep the records which are within the time interval range.

library(dplyr)
library(lubridate)

left_join(DF2, DF1, by = "country_code") %>% 
      filter(event_date %within% office_interval)

#  event_id country_code event_date leader_id                office_interval
#1        1            1 1901-01-02      Adam 1900-01-01 UTC--1905-01-01 UTC
#2        1            2 1920-01-02   Charlie 1920-01-01 UTC--1925-01-01 UTC
#3        2            2 1921-01-02   Charlie 1920-01-01 UTC--1925-01-01 UTC
#4        3            1 1911-01-02       Bob 1910-01-01 UTC--1915-01-01 UTC
#5        3            3 1941-02-02     Edgar 1940-01-01 UTC--1945-01-01 UTC

edited Jan 20 '20 at 13:34

answered Jan 20 '20 at 13:29

Ronak Shah

377,200
20
156
213

This seems to be on the right track, although when I apply it to my dataset the number of rows increases from 5,000 to 78,000. I dont know why it is adding any new rows when there are no overlapping office_intervals within each unique country_code – BobC Jan 20 '20 at 13:55
@BobC Try `left_join(DF2, DF1, by = "country_code") %>% filter(event_date %within% office_interval) %>% distinct()` Or do you need an `inner_join` ? – Ronak Shah Jan 20 '20 at 13:57
okay my fault, I got two datasets switched up. My original dataset has observations by the leader-year, so there are multiple rows from which to x can join. The one I presented here has only one leader per observation so the answers here are correct for the latter dataset. For the leader-year data, I simply just added "year" to the left join and it seemed to work. Thanks. – BobC Jan 20 '20 at 14:43

score 0 · Answer 3 · answered Jan 20 '20 at 13:29

This should also work:

# add start and end date
DF1$start_date <- substr(DF1$office_interval, 1, 10)
DF1$end_date <- substr(DF1$office_interval, 17, 26)

# merge dataframes
DF2 <- merge(x = DF2, y = DF1, by.x = "country_code", by.y = "country_code")

# filter for correct times
DF2 <- DF2[(DF2$event_date >= DF2$start_date & DF2$event_date <= DF2$end_date),]

# select columns
DF2[1:4]

merging two datasets based on common id and date within interval range

3 Answers3