I need to download a lot of files, but I want to process only 3 files at the same time. I'm receiving the next file URL from a Tokio MPSC channel. I need to cover the case when I'm already downloading 2 files and waiting for a new one.
How can I wait for one of 3 futures and extend a previous select_all
?
A very simplified example: playground
use futures::{
future::{select, select_all, Either},
FutureExt,
};
use tokio::time::{delay_for, Duration};
async fn exec(val: usize) {
println!("Started {}", val);
delay_for(Duration::from_millis(100)).await;
println!("Executed {}", val);
}
async fn get_next() -> usize {
use rand::Rng;
delay_for(Duration::from_millis(100)).await;
rand::thread_rng().gen_range(0, 10)
}
#[tokio::main]
async fn main() {
let mut tasks = Vec::new();
for _ in 0..10 {
if tasks.len() == 0 {
let next = get_next().await;
tasks.push(tokio::spawn(exec(next)));
continue;
}
let all = select_all(tasks.drain(..));
let next = get_next().boxed();
match select(all, next).await {
Either::Left(((_, _, left), _)) => tasks = left,
Either::Right((a, _other)) => {
// How can I get tasks futures from 'other'?
// So on the next iteration, I'll be able to do select with an additional task.
tasks.push(tokio::spawn(exec(a)));
}
};
}
}