`r_make()` and `make()` not consistent in r-drake

Question

I'm using RStudio for the work in question. I used to use drake::make() ignoring the prompt to use r_make() till yesterday when I decided to give it a try.

Now, I'm in a bit of a pickle. Not sure what I've done / if I have found a bug.

My project plan is not large, but the target builds are large (10-20 min data pulls each). I've run it end-to-end using make() from the RStudio console. Everything is up-to-date. Even if I open a separate R session in terminal, and run outdated(), I don't see anything outdated.

However, running r_make() from the Rstudio console, or from R in a separate terminal results in many targets out of date.

And no matter how many times I run r_make(), these objects remain un-updated.

I don't know how to proceed. I'd like to keep using r_make(), but I'm stuck.

Update

Reproducible example at https://gitlab.com/rsangole/drake-test

Update 2

Running @landau's reprex example, output is here. It has worked correctly, as Will obtained with this script: https://gitlab.com/rsangole/drake-test/snippets/1894934

However...

If I run https://gitlab.com/rsangole/drake-test/blob/master/make.R in interactive mode in RStudio, I can reproduce my condition. To reproduce:

Open new Rstudio, clean env
Download reproducible example in https://gitlab.com/rsangole/drake-test
Run https://gitlab.com/rsangole/drake-test/blob/master/make.R
Expect output as in https://gitlab.com/rsangole/drake-test/snippets/1894937

Same issue even after updating drake to latest bleeding edge version

Also able to replicate issue if I run make.R via R in zsh thereby bypassing RStudio completely.

Would you share the code that reproduces what you are seeing? The reprex package is helpful for this. — landau, Sep 13 '19 at 22:48
Let me build a reproducible example and share. I haven't checked if this is unique to my project or can be reproduced either. Checking... — Rahul, Sep 13 '19 at 23:10
@landau - updated with link to reproducible example uploaded to gitlab — Rahul, Sep 14 '19 at 00:03
Are you able to provide a reproducible example that does not require downloading your package? Keeping it reproducible within the confines of *this question* can be a really good thing. — r2evans, Sep 14 '19 at 00:18
@r2evans I actually created that gitlab repo particularly just to make the problem reproducible within using a very light wt example. The repo isn't my original codebase (which is massive). — Rahul, Sep 14 '19 at 07:48
The only part I kept consistent is all the packages within `R/packages.R` just in case that's the root cause of something creating a conflict... — Rahul, Sep 14 '19 at 07:51
@Rahul, I tried steps 1-4 in your second update, and I still cannot reproduce what you are seeing. The only snag is that I had to comment out `library(hubbler)` since I could not find this package. Where do I download it? Also, when you run the example yourself, what happens when you remove `lock_envir = FALSE`? — landau, Sep 15 '19 at 11:45
@landau you can ignore that one. It's an internally developed private lib I have. I shouldn't commented it out for you. I'll check what happens with `lock_envir`. I also have a custom `.Rprofile` which adds some functions to the global env. Think that is contributing to anything? — Rahul, Sep 16 '19 at 01:56
Yes, `.Rprofile` is worth investigating. Good thinking. If you see a good place to mention `.Rprofile`s in `drake`'s docs, feel free to submit a pull request. — landau, Sep 16 '19 at 02:19

landau · Answer 1 · 2019-09-14T11:25:22.527

I created a reprex using https://gitlab.com/rsangole/drake-test, and I could not reproduce what you are seeing. Are you using drake differently than in the code below?

library(drake)

# Write script files

package_code <- quote({
  library(data.table)
  library(drake)
})

function_code <- quote(
  make_data <- function(download_date){
    data.table(
      dt = download_date,
      x = runif(n = 1e5),
      y = sample(letters,size = 1e5,replace = T)
    )
  }
)

plan_code <- quote(
  plan <- drake_plan(
    download_date = "2019-09-08",
    x = make_data(download_date)
  )
)

writeLines(deparse(package_code), "packages.R")
writeLines(deparse(function_code), "functions.R")
writeLines(deparse(plan_code), "plan.R")

# Run make()

source("packages.R")
source("functions.R")
source("plan.R")

make(
  plan,
  verbose = 2,
  parallelism = "future",
  jobs = 4L,
  lock_envir = FALSE
)
#> 
target download_date
#> 
target x

make(
  plan,
  verbose = 2,
  parallelism = "future",
  jobs = 4L,
  lock_envir = FALSE
)
#> 
All targets are already up to date.

config <- drake_config(plan)
outdated(config)
#> character(0)

# Write _drake.R

drake_code <- quote({
  source("packages.R")
  source("functions.R")
  source("plan.R")
  drake_config(
    plan, 
    verbose = 2, 
    jobs = 3L, 
    lock_envir = FALSE, 
    parallelism = "future"
  )
})

writeLines(deparse(drake_code), "_drake.R")

# Run r_make()

r_make()
#> 
#> ..
[34mAll targets are already up to date.[39m

r_make()
#> 
#> .
[34mAll targets are already up to date.[39m

r_outdated()
#> 
#> .
#> character(0)

^{Created on 2019-09-13 by the reprex package (v0.3.0)}

devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 3.6.1 (2019-07-05)
#>  os       Ubuntu 18.04.2 LTS          
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language                             
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2019-09-13                  
#> 
#> ─ Packages ──────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 3.6.1)
#>  backports     1.1.4   2019-04-10 [1] CRAN (R 3.6.1)
#>  base64url     1.4     2018-05-14 [1] CRAN (R 3.6.1)
#>  callr         3.3.1   2019-07-18 [1] CRAN (R 3.6.1)
#>  cli           1.1.0   2019-03-19 [1] CRAN (R 3.6.1)
#>  codetools     0.2-16  2018-12-24 [1] CRAN (R 3.6.1)
#>  crayon        1.3.4   2017-09-16 [1] CRAN (R 3.6.1)
#>  data.table  * 1.12.2  2019-04-07 [1] CRAN (R 3.6.1)
#>  desc          1.2.0   2018-05-01 [1] CRAN (R 3.6.1)
#>  devtools      2.2.0   2019-09-07 [1] CRAN (R 3.6.1)
#>  digest        0.6.20  2019-07-04 [1] CRAN (R 3.6.1)
#>  drake       * 7.6.1   2019-08-19 [1] CRAN (R 3.6.1)
#>  DT            0.8     2019-08-07 [1] CRAN (R 3.6.1)
#>  ellipsis      0.2.0.1 2019-07-02 [1] CRAN (R 3.6.1)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 3.6.1)
#>  filelock      1.0.2   2018-10-05 [1] CRAN (R 3.6.1)
#>  fs            1.3.1   2019-05-06 [1] CRAN (R 3.6.1)
#>  future        1.14.0  2019-07-02 [1] CRAN (R 3.6.1)
#>  globals       0.12.4  2018-10-11 [1] CRAN (R 3.6.1)
#>  glue          1.3.1   2019-03-12 [1] CRAN (R 3.6.1)
#>  highr         0.8     2019-03-20 [1] CRAN (R 3.6.1)
#>  htmltools     0.3.6   2017-04-28 [1] CRAN (R 3.6.1)
#>  htmlwidgets   1.3     2018-09-30 [1] CRAN (R 3.6.1)
#>  igraph        1.2.4.1 2019-04-22 [1] CRAN (R 3.6.1)
#>  knitr         1.24    2019-08-08 [1] CRAN (R 3.6.1)
#>  listenv       0.7.0   2018-01-21 [1] CRAN (R 3.6.1)
#>  magrittr      1.5     2014-11-22 [1] CRAN (R 3.6.1)
#>  memoise       1.1.0   2017-04-21 [1] CRAN (R 3.6.1)
#>  pillar        1.4.2   2019-06-29 [1] CRAN (R 3.6.1)
#>  pkgbuild      1.0.5   2019-08-26 [1] CRAN (R 3.6.1)
#>  pkgconfig     2.0.2   2018-08-16 [1] CRAN (R 3.6.1)
#>  pkgload       1.0.2   2018-10-29 [1] CRAN (R 3.6.1)
#>  prettyunits   1.0.2   2015-07-13 [1] CRAN (R 3.6.1)
#>  processx      3.4.1   2019-07-18 [1] CRAN (R 3.6.1)
#>  ps            1.3.0   2018-12-21 [1] CRAN (R 3.6.1)
#>  R6            2.4.0   2019-02-14 [1] CRAN (R 3.6.1)
#>  Rcpp          1.0.2   2019-07-25 [1] CRAN (R 3.6.1)
#>  remotes       2.1.0   2019-06-24 [1] CRAN (R 3.6.1)
#>  rlang         0.4.0   2019-06-25 [1] CRAN (R 3.6.1)
#>  rmarkdown     1.15    2019-08-21 [1] CRAN (R 3.6.1)
#>  rprojroot     1.3-2   2018-01-03 [1] CRAN (R 3.6.1)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 3.6.1)
#>  storr         1.2.1   2018-10-18 [1] CRAN (R 3.6.1)
#>  stringi       1.4.3   2019-03-12 [1] CRAN (R 3.6.1)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 3.6.1)
#>  testthat      2.2.1   2019-07-25 [1] CRAN (R 3.6.1)
#>  tibble        2.1.3   2019-06-06 [1] CRAN (R 3.6.1)
#>  txtq          0.1.5   2019-08-19 [1] CRAN (R 3.6.1)
#>  usethis       1.5.1   2019-07-04 [1] CRAN (R 3.6.1)
#>  withr         2.1.2   2018-03-15 [1] CRAN (R 3.6.1)
#>  xfun          0.9     2019-08-21 [1] CRAN (R 3.6.1)
#>  yaml          2.2.0   2018-07-25 [1] CRAN (R 3.6.1)
#> 
#> [1] /home/landau/R/R-3.6.1/library

Update

Please run this reprex yourself and post the result:

reprex::reprex({
library(drake)

# Write script files

package_code <- c(
  "library(data.table)",
  "library(drake)"
)

function_code <- c(
  "make_data <- function(download_date){",
  "  data.table(",
  "    dt = download_date,",
  "    x = runif(n = 1e5),",
  "    y = sample(letters,size = 1e5,replace = T)",
  "  )",
  "}"
)

plan_code <- c(
  "plan <- drake_plan(",
  "  download_date = \"2019-09-08\",",
  "  x = make_data(download_date)",
  ")"
)

drake_code <- c(
  "source(\"packages.R\")",
  "source(\"functions.R\")",
  "source(\"plan.R\")",
  "drake_config(",
  "  plan,",
  "  verbose = 2,",
  "  jobs = 3L,",
  "  lock_envir = FALSE,",
  "  parallelism = \"future\",",
  "  console_log_file = \"r_make.log\"",
  ")"
)

writeLines(package_code, "packages.R")
writeLines(function_code, "functions.R")
writeLines(plan_code, "plan.R")
writeLines(drake_code, "_drake.R")

# Check that we wrote the scripts properly

cat(readLines("packages.R"), sep = "\n")

cat(readLines("functions.R"), sep = "\n")

cat(readLines("plan.R"), sep = "\n")

cat(readLines("_drake.R"), sep = "\n")

# Is there already .drake/ cache somewhere? 

list.files()

find_cache()


# If so, this reprex is not clean.

for (i in seq_len(10)) {
  clean(destroy = TRUE)
}

# Get ready

source("packages.R")
source("functions.R")
source("plan.R")
config <- drake_config(plan)

# Run make() and r_make()

make(
  plan,
  verbose = 2,
  parallelism = "future",
  jobs = 4L,
  lock_envir = FALSE,
  console_log_file = "make.log"
)

outdated(config)

r_make()

r_outdated()

make(
  plan,
  verbose = 2,
  parallelism = "future",
  jobs = 4L,
  lock_envir = FALSE,
  console_log_file = "make.log"
)

outdated(config)

r_make()

r_outdated()

# Check the log files to see what triggered the targets.

cat(readLines("make.log"), sep = "\n")

cat(readLines("r_make.log"), sep = "\n")
}, si = TRUE, venue = "so")

Thank you @landau. Interesting. I'll dig deeper and see how I can make it reproducible for you. — Rahul, Sep 14 '19 at 07:48
I was able to reproduce my problem even with your codebase. I can keep alternating between `r_make()` and `make()` and each ones makes the other "outdated". Check update-3 for the image showing this. — Rahul, Sep 14 '19 at 08:01
are `r_make()` and `make()` supposed to work interchangeably, as I'm assuming them to be? i.e. if I run `make()` , `r_outdated()` should also detect no new targets? — Rahul, Sep 14 '19 at 08:19
It is difficult for me to get to the bottom of this using screenshots that start with `drake_config()` and `make()`. I included the code for a [`reprex`](https://github.com/tidyverse/reprex) in the update to my answer. Please run that call to `reprex::reprex()` and post *all* the results it generates. — landau, Sep 14 '19 at 11:27
The [`reprex`](https://github.com/tidyverse/reprex) will not only show the results, but also how the scripts are generated. It also makes sure you begin with a clean file system and clean environment. This is key for a truly reproducible example. — landau, Sep 14 '19 at 11:28
I also recommend reading https://reprex.tidyverse.org/articles/reprex-dos-and-donts.html and https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example/16532098 to learn more about reproducible examples. In your case, I would say please start your examples right from the beginning with an empty workspace and new R session (which the `reprex` package does automatically for you). — landau, Sep 14 '19 at 11:35
Gotcha. Thanks @landau. I'll rerun and edit my original post using `reprex` shortly. — Rahul, Sep 15 '19 at 04:37
Original post updated. Reprex output attached in snippet. Also added more details... — Rahul, Sep 15 '19 at 06:05
Just letting you guys know that both https://gitlab.com/rsangole/drake-test and the reprex of @landau answer (are they identical?) work for me without problems. — pat-s, Sep 18 '19 at 11:38
Thanks for checking @pat-s. I suspect the custom `.Rprofile` I have which isn't in the reprex. I'll investigate this week and revert... — Rahul, Sep 19 '19 at 00:23

score 0 · Accepted Answer · answered Oct 10 '19 at 03:48

0

I found the issue. I was using an MRAN snapshot of '2019-06-01'-ish. Upgrading all packages to MRAN of Oct-10 fixed everything.

answered Oct 10 '19 at 03:48

Rahul

2,579
1
13
22

`r_make()` and `make()` not consistent in r-drake

Update

Update 2

However...

2 Answers2

Update