Updating raster values in foreach loops in R? (raster time-series NA imputation)

Question

I am trying to impute NA values in a time series raster. Here is an reproducible example of my data:

library(raster)
library(rgdal)
library(doParallel)
library(foreach)

r1 <- r2 <- r3 <- r4 <- r5 <- raster(nrow=100, ncol=100)
values(r1) <- runif(ncell(r1))
values(r2) <- runif(ncell(r2))
values(r3) <- runif(ncell(r3))
values(r4) <- runif(ncell(r4))
values(r5) <- runif(ncell(r5))

s <- stack(r1, r2, r3, r4, r5)
time_series <- brick(s)
time_series[1, 30][2] <- NA
time_series[3, 20][3] <- NA
time_series[5, 10][5] <- NA
time_series[8, 40][4] <- NA

There are packages such as gapfill but I found them too slow for my task. I found another method here as in the answer: https://gis.stackexchange.com/questions/279354/ndvi-time-series-with-missing-values

by: https://gis.stackexchange.com/users/8520/jeffrey-evans

I want to convert the for loop to foreach so I can calculate it for larger images. Here is the code with for loops:

impute.loess <- function(y, x.length = NULL, s = 0.80, 
                         smooth.data = FALSE, ...) {
  if(is.null(x.length)) { x.length = length(y) }
  options(warn = -1)
  x <- 1:x.length
  if (all(is.na(y))) {
    return(y)
  } else {
    p <- loess(y ~ x, span = s, data.frame(x = x, y = y))
    if(smooth.data == TRUE) {
      y <- predict(p, x)
    } else {
      na.idx <- which( is.na(y) )
      if( length(na.idx) > 1 ) {
        y[na.idx] <- predict(p, data.frame(x = na.idx))
      }
    }   
    return(y)
  }
}

time_series_new <- time_series

time_series_new[] <- NA

for (rl in 1:nrow(time_series)) {
  v <- getValues(time_series, rl, 1)
  time_series_new[rl,] <- as.matrix(t(apply(v, MARGIN=1, FUN=impute.loess)))
}

Foreach alternative I tried is this:

time_series_new2 <- time_series

time_series_new2[] <- NA

cl <- parallel::makeCluster(detectCores()-1)
doParallel::registerDoParallel(cl)

time_series_new2 <- foreach (rl = 1:nrow(time_series),
                             .packages = "raster",
                             .combine = 'rbind') %dopar% {
                        v <- getValues(time_series, rl, 1)
                        time_series_new[rl,] <- as.matrix(t(apply(v,
                                                MARGIN=1, FUN=impute.loess)))
}

parallel::stopCluster(cl)

However, here is the difference:

> class(time_series_new)
[1] "RasterBrick"
attr(,"package")
[1] "raster"

> class(time_series_new2)
[1] "matrix"

If I don't assign foreach loop to an object, it just export the outcome. I want an updated raster object at the end but couldn't find a solution to my problem.

I couldn't find how to set matrix values raster object - setvalues didn't work maybe since the dimensions are different as:

> dim(time_series_new)
[1] 100 100   5

> dim(time_series_new2)
[1] 10000     5

I know foreach loop works differently. Is there a way to update time_series_new2 object inside foreach loop so that I can have an updated raster object at the end?

EDIT:

setValues() actually works! as:

time_series_new3 <- time_series

time_series_new3[] <- NA #empty raster object

time_series_new3 <- setValues(time_series_new3, time_series_new2) #filled with matrix rendered from foreach loop

> time_series_new3
class      : RasterBrick 
dimensions : 100, 100, 10000, 5  (nrow, ncol, ncell, nlayers)
resolution : 3.6, 1.8  (x, y)
extent     : -180, 180, -90, 90  (xmin, xmax, ymin, ymax)
crs        : +proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0 
source     : memory
names      :      layer.1,      layer.2,      layer.3,      layer.4,      layer.5 
min values : 1.468023e-04, 3.525158e-04, 9.689084e-05, 5.349121e-05, 4.214607e-05 
max values :    0.9999564,    0.9999854,    0.9997795,    0.9999780,    0.9997880 

> time_series_new2
class      : RasterBrick 
dimensions : 100, 100, 10000, 5  (nrow, ncol, ncell, nlayers)
resolution : 3.6, 1.8  (x, y)
extent     : -180, 180, -90, 90  (xmin, xmax, ymin, ymax)
crs        : +proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0 
source     : memory
names      :      layer.1,      layer.2,      layer.3,      layer.4,      layer.5 
min values : 1.468023e-04, 3.525158e-04, 9.689084e-05, 5.349121e-05, 4.214607e-05 
max values :    0.9999564,    0.9999854,    0.9997795,    0.9999780,    0.9997880

> all.equal(time_series_new2, time_series_new3)
[1] TRUE

Still, I would like to know about updating in foreach.

Answer 1

In the foreach loop, you don't have the luxury of the side effect of updating the rasterBrick time_series_new. That is, time_series_new knows what it is - a raster type of object. The rbind combine will coerce non-data.frames into matrices. That's how 100 x 100 x 5 becomes 10000 x 5.

I assume you are going to parallel computing due to the slowness of the for loop. If that's the case, I recommend approaching the problem differently, especially if there aren't a lot of missing values.

We can first see how many rows actually have missing data:

missing_dat_rows <- which(is.na(getValues(time_series)) == T, arr.ind = T)[, 1]
missing_dat_rows <- unique(missing_dat_rows)

missing_dat_rows
#[1]  30 220 740 410

So instead of looping through 10,000 results, we now can concentrate on these 4 results.

time_series3 <- time_series
for (mis_row in missing_dat_rows) {
  values(time_series3)[mis_row, ] <- impute.loess(getValues(time_series3)[mis_row, ])
}

Unfortunately, I couldn't get the impute.loess() function to return values for me. I made a couple small big changes that may be helpful as well if you want to continue your loop approach:

impute.loess <- function(y, x.length = NULL, s = 0.80, 
                         smooth.data = FALSE, ...) {
  if(is.null(x.length)) { x.length = length(y) }
  options(warn = -1)

  x <- 1:x.length
  if (all(is.na(y))| all(!is.na(y))) { #added the or statement - I don't think we want to do this if there are no missing values.
    return(y)
  } else {
    p <- loess(y ~ x, span = s, data.frame(x = x, y = y))
    if(smooth.data == TRUE) {
      y <- predict(p, x)
    } else {
      na.idx <- which( is.na(y) )
      # if( length(na.idx) > 1 ) { #commented out - I feel as though we should be replacing all NAs
        y[na.idx] <- predict(p, data.frame(x = na.idx))
      # }
    }   
    return(y)
  }
}