Stack Overflow
Stack Overflow

Questions tagged [furrr]

This tag addresses the R package furrr intended to simplify the combination of purrr’s family of mapping functions.

This tag addresses the R package furrr intended to simplify the combination of purrr’s family of mapping functions.

68 questions
6
votes
0 answers

Progress bar in Shiny using furrr and future

I am developing a Shiny app where I split time consuming calculation to multiple session using furrr:: and future. I would like to update the progress bar after every (or every n) calculations. The below example I found updates the progress bar but…
capron82
  • 61
  • 1
6
votes
0 answers

How to rate-limit parallel API requests in R/future/furrr

I have to retrieve a large dataset from a web API (NCBI entrez) that limits me to a certain number of requests per second, say 10 (the example code will limit you to three without an API key). I'm using furrr's future_* functions to parallelize the…
Luther Blissett
  • 373
  • 1
  • 10
6
votes
1 answer

Nested furrr::future_map?

Is there a way to configure furrr::future_map that would allow a nested use case ? Consider the following code : library(furrr) library(tictoc) # The problem is easier to reason about if you take N # smaller than your number of cores, and M big. N…
lrnv
  • 1,038
  • 8
  • 19
6
votes
1 answer

furrr doesn't find own package

I am currently working on a package, let's say it is called myPack. I have a function called myFunc1 and another function called myFunc2, which looks something like that: myFunc2 <- function(x, parallel = FALSE) { if(parallel)…
user11143533
  • 79
  • 3
5
votes
1 answer

furrr / purrr progressr progress bar not at all synched with the progress of the computation

I want to create a function that takes a function and applies it once for every row in a tibble with arguments stored in the correspondingly named columns of the tibble I realize that this sounds a bit odd, but I want the user facing function /…
Fredrik Karlsson
  • 485
  • 8
  • 21
5
votes
1 answer

Why is `furrr::future_map_int()` slower than `purrr::map_int()` when I use `dplyr::mutate()`?

I have a tibble that includes a list-column with vectors inside. I want to create a new column that accounts for the length of each vector. Since this dataset is large (3M rows), I thought to shave off some processing time using the furrr package.…
Emman
  • 3,695
  • 2
  • 20
  • 44
5
votes
2 answers

How to do faster list-column operations inside data.table

Due to memory (and speed) issues, I was hoping to do some computations inside a data.table instead of doing them outside it. The following code has 100.000 rows, but I'm working with 40 million rows. library(tictoc) library(data.table) # version…
Telaroz
  • 173
  • 1
  • 7
4
votes
0 answers

Understanding memory usage and performance of `furrr::future_apply`

I am facing some issues parallelizing processes with furrr::future_apply (see Optimizing memory usage in applying a furrr function to a large list of tibbles: experiencing unexpected memory increase, and memory not being released in future with…
alexon
  • 61
  • 3
4
votes
2 answers

R: asynchronous parallel lapply

The simplest way I've found so far to use a parallel lapply in R was through the following example code: library(parallel) library(pbapply) cl <- makeCluster(10) clusterExport(cl = cl, {...}) clusterEvalQ(cl = cl, {...}) results <- pblapply(1:100,…
runr
  • 1,142
  • 1
  • 9
  • 25
3
votes
0 answers

parallel computation with furrr on sf objects : future_map and map don't handle sf object the same way

I'm working on a sf points object. I wrote a code to do some grouping and computing mean distance to the nearest neighbor and count by cities. When I do all the processus with dplyr and purrr, it's working nicely (but it takes some time because my…
pgourdon
  • 139
  • 7
3
votes
0 answers

furrr with rTorch in multisession

I want to use rTorch in a furrr "loop". A minimal example seems to be: library(rTorch) torch_it <- function(i) { #.libPaths("/User/homes/mreichstein/.R_libs_macadamia4.0/") #require(rTorch) cat("torch is: "); print(torch) out <-…
MR_MPI-BGC
  • 265
  • 3
  • 11
3
votes
1 answer

Is there a way to modify the cluster resources allocated futures when using a furrr function?

I'm using the future, future.batchtools and furrr packages within my R script, in order to run code on a slurm-managed remote machine. When I define my future topology, I pass a template file to future.bacthtools::batchtools_slurm() and define the…
Johannes Koch
  • 71
  • 2
2
votes
0 answers

Optimizing memory usage in applying a furrr function to a large list of tibbles: experiencing unexpected memory increase

I am currently working on a task that involves applying a function to a fairly extensive list of tibbles, comprising approximately 30,000 elements. The code I'm using is as follows: plan(multisession, workers=20) hpar$input_df %>% group_by(key)…
alexon
  • 61
  • 3
2
votes
1 answer

Ensure reproducibility across `purrr::map()` and `furrr::future_map()`

I am running simulations where some computing should be parallelized and some should not. I am trying to figure out how to ensure reproducibility across purrr::map() and furrr::future_map() so that they yield the same result. For some reason, I…
Dan Chaltiel
  • 7,811
  • 5
  • 47
  • 92
2
votes
0 answers

Saving patchwork ggplots with furrr future map only saves one of the plots

I am trying to save a list of patchworked ggplots (ie ggplots that have been wrapped together into 1 plot using the package patchwork). My real data is creating several hundred plots so I would like to use furrr to speed it up. Using…
Sarah
  • 3,022
  • 1
  • 19
  • 40
1
2 3 4 5