0

I've been looking and trying different methods to concatenate a huge number of strings, but all of them have not worked after 15+ minutes of waiting. Is there a faster or more efficient way to do this? Below I have the code to generate 20,000 random strings, find unique combinations using expand.grid, then paste them together. The last line keep crashing my R Studio.

myFun <- function(n = 5000) {
  a <- do.call(paste0, replicate(5, sample(LETTERS, n, TRUE), FALSE))
  paste0(a, sprintf("%04d", sample(9999, n, TRUE)), sample(LETTERS, n, TRUE))
}

x <- myFun(20000)
y <- do.call(paste0, expand.grid(x, x))
purple1437
  • 313
  • 1
  • 8

1 Answers1

1

Concatenating a large number of strings can be a computationally expensive operation, particularly if the number of strings is very large. One possible way to make this more efficient is to use a data structure that is optimized for string concatenation, such as a StringBuilder in languages like Java and C#, or a StringIO object in Python.

In R, you can use the paste() or paste0() function to concatenate strings, but both function might not be efficient when you are working with a very large number of strings. A more efficient approach would be to use the str_c() function from the stringr package. The str_c() function uses a different algorithm for concatenating strings that can be much faster when working with large datasets.

Additionally, you could try to avoid concatenation altogether. You should think about pre-allocating the strings of the desired size and then filling them in.

Another thing to consider when working with such large datasets is the memory usage, it could be that you are running out of memory. Therefore you could consider breaking the process into chunks and process them separately.

Lastly, you can check the usage of foreach and parallel libraries in R, which could be use to parallelize the computation.

Pablo Rodriguez
  • 233
  • 1
  • 10