Bind residuals to input dataset with missing values

Question

I am looking for a method to bind lm residuals to an input dataset. The method must add NA for missing residuals and the residuals should correspond to the proper row.

Sample data:

N <- 100 
Nrep <- 5 
X <- runif(N, 0, 10) 
Y <- 6 + 2*X + rnorm(N, 0, 1) 
X[ sample(which(Y < 15), Nrep) ] <- NA
df <- data.frame(X,Y)

residuals(lm(Y ~ X,data=df,na.action=na.omit))

Residuals should be bound to df.

Similar questions [here](http://stackoverflow.com/q/6882709/684229) and [here](http://stats.stackexchange.com/questions/11000/how-does-r-handle-missing-values-in-lm). — Tomas, Jul 31 '13 at 18:26

score 10 · Answer 1 · edited Apr 13 '17 at 12:44

10

Simply change the na.action to na.exclude:

residuals(lm(Y ~ X, data = df, na.action = na.exclude))

na.omit and na.exclude both do casewise deletion with respect to both predictors and criterions. They only differ in that extractor functions like residuals() or fitted() will pad their output with NAs for the omitted cases with na.exclude, thus having an output of the same length as the input variables.

(this is the best solution found here)

edited Apr 13 '17 at 12:44

Community

1
1

answered Jul 31 '13 at 18:23

Tomas

57,621
49
238
373

This is the general solution you're looking for, the one that works with missings in any number of predictors or DV, with lm and lme4. – Ruben Dec 08 '14 at 11:40

score 2 · Answer 2 · answered Dec 02 '12 at 19:19

2

Using merge, or join.

N <- 100 
Nrep <- 5 
X <- runif(N, 0, 10) 
Y <- 6 + 2*X + rnorm(N, 0, 1) 
X[ sample(which(Y < 15), Nrep) ] <- NA
df <- data.frame(X,Y)

df$id <- rownames(df)

res <- residuals(lm(Y ~ X,data=df,na.action=na.omit))
tmp <- data.frame(res=res)
tmp$id <- names(res)

merge(df,tmp,by="id",sort=FALSE,all.x=TRUE)

If you need to maintain the order. Use join() from the plyr package:

library(plyr) 
join(df,tmp)

answered Dec 02 '12 at 19:19

Brandon Bertelsen

43,807
34
160
255

couldn't this code be simplified by [merging by row names](http://stackoverflow.com/a/7739757/684229)? – Tomas Jul 31 '13 at 18:08
There is much much simpler solution, see my answer – Tomas Jul 31 '13 at 18:24

score 0 · Answer 3 · answered Dec 02 '12 at 19:15

0

This maybe could be solution, but, first, you do not need c() in data.frame

df <- data.frame(X,Y)
df$Res[!is.na(X)]<-residuals(lm(Y ~ X,data=df,na.action=na.omit))

answered Dec 02 '12 at 19:15

Didzis Elferts

95,661
14
264
201

This duplicates residuals. Rather than appending NA – Brandon Bertelsen Dec 02 '12 at 19:17
I've removed the `c()` in `data.frame` – metasequoia Dec 02 '12 at 19:17
What if Y is NA? What if another predictor variables is NA? Not very robust to this, thus probably not a way to go. – Tomas Jul 31 '13 at 18:09

score 0 · Accepted Answer · answered Dec 02 '12 at 19:44

0

"[<-"(df, !is.na(df$X), "res", residuals(lm(Y ~ X,data=df,na.action=na.omit)))

will do the trick.

answered Dec 02 '12 at 19:44

Sven Hohenstein

80,497
17
145
168

Can you explain this? What is `"[<-"`? – Brandon Bertelsen Dec 02 '12 at 20:08
@BrandonBertelsen The function `"[<-"(x1, x2, x3, x4)` is similar to `x1[x2, x3] <- x4` but leaves `x1` unchanged and returns a new object. – Sven Hohenstein Dec 03 '12 at 07:02

score 0 · Answer 5 · answered Jul 19 '17 at 18:56

N <- 100 
Nrep <- 5 
X <- runif(N, 0, 10) 
Y <- 6 + 2*X + rnorm(N, 0, 1) 
X[ sample(which(Y < 15), Nrep) ] <- NA
df <- data.frame(X,Y)

R.all=as.numeric(rep(NA,length(X)))  # numeric vector with missing values
res=residuals(lm(Y ~ X,data=df,na.action=na.omit))  
i=as.numeric(names(res)) # vector locations of non-missing residuals
R.all[i]=res;R.all     # assign residuals to their correct positions.

Bind residuals to input dataset with missing values

5 Answers5

Linked