fast way to separate list of list into two lists

Question

I have got quite a good experience with C programming and I am used to think in terms of pointers, so I can get good performance when dealing with huge amount of datas. It is not the same with R, which I am still learning.

I have got a file with approximately 1 million lines, separated by a '\n' and each line has got 1, 2 or more integers inside, separated by a ' '. I have been able to put together a code which reads the file and put everything into a list of lists. Some lines can be empty. I would then like to put the first number of each line, if it exists, into a separated list, just passing over if a line is empty, and the remaining numbers into a second list.

The code I post here is terribly slow (it has been still running since I started wrote this question so now I killed R), how can I get a decent speed? In C this would be done instantly.

graph <- function() {
    x <- scan("result", what="", sep="\n")
    y <- strsplit(x, "[[:space:]]+") #use spaces for split number in each line
    y <- lapply(y, FUN = as.integer) #convert from a list of lists of characters to a list of lists of integers
    print("here we go")
    first <- c()
    others <- c()
    for(i in 1:length(y)) {
        if(length(y[i]) >= 1) { 
            first[i] <- y[i][1]
        }
        k <- 2;
        for(j in 2:length(y[i])) {
            others[k] <- y[i][k]
            k <- k + 1
        }
    }

In a previous version of the code, in which each line had at least one number and in which I was interested only in the first number of each line, I used this code (I read everywhere that I should avoid using for loops in languages like R)

yy <- rapply(y, function(x) head(x,1))

which takes about 5 second, so far far better than above but still annoying if compared to C.

EDIT this is an example of the first 10 lines of my file:

42 7 31 3 
23 1 34 5 


1 
-23 -34 2 2 

42 7 31 3 31 4 

1

Answer 1

Base R versus purrr

your_list <- rep(list(list(1,2,3,4), list(5,6,7), list(8,9)), 100)

microbenchmark::microbenchmark(
  your_list %>% map(1),
  lapply(your_list, function(x) x[[1]])
)
Unit: microseconds
                                  expr       min        lq       mean    median         uq       max neval
                  your_list %>% map(1) 22671.198 23971.213 24801.5961 24775.258 25460.4430 28622.492   100
 lapply(your_list, function(x) x[[1]])   143.692   156.273   178.4826   162.233   172.1655  1089.939   100

---

microbenchmark::microbenchmark(
  your_list %>% map(. %>% .[-1]),
  lapply(your_list, function(x) x[-1])
)
Unit: microseconds
                                 expr     min       lq      mean   median       uq      max neval
       your_list %>% map(. %>% .[-1]) 916.118 942.4405 1019.0138 967.4370 997.2350 2840.066   100
 lapply(your_list, function(x) x[-1]) 202.956 219.3455  264.3368 227.9535 243.8455 1831.244   100

purrr isn't a package for performance, just convenience, which is great but not when you care a lot about performance. This has been discussed elsewhere.

---

By the way, if you are good in C, you should look at package Rcpp.

Answer 2

try this:

your_list <- list(list(1,2,3,4),
     list(5,6,7),
     list(8,9))

library(purrr)

first <- your_list %>% map(1)
# [[1]]
# [1] 1
# 
# [[2]]
# [1] 5
# 
# [[3]]
# [1] 8

other <- your_list %>% map(. %>% .[-1])    
# [[1]]
# [[1]][[1]]
# [1] 2
# 
# [[1]][[2]]
# [1] 3
# 
# [[1]][[3]]
# [1] 4
# 
# 
# [[2]]
# [[2]][[1]]
# [1] 6
# 
# [[2]][[2]]
# [1] 7
# 
# 
# [[3]]
# [[3]][[1]]
# [1] 9

Though you might want the following, as it seems to me those numbers would be better stored in vectors than in lists:

your_list %>% map(1) %>% unlist # as it seems map_dbl was slow
# [1] 1 5 8
your_list %>% map(~unlist(.x[-1]))
# [[1]]
# [1] 2 3 4
# 
# [[2]]
# [1] 6 7
# 
# [[3]]
# [1] 9

Answer 3

Indeed, coming from C to R will be confusing (it was for me). What helps for performance is understanding that primitive types in R are all vectors implemented in highly optimized, natively-compiled C and Fortran, and you should aim to avoid loops when there's a vectorized solution available.

That said, I think you should load this as a csv via read.csv(). This will provide you with a dataframe with which you can perform vector-based operations.

For a better understanding, a concise (and humorous) read is http://www.burns-stat.com/pages/Tutor/R_inferno.pdf.

Answer 4

I would try to use stringr package. Something like this:

set.seed(3)
d <- replicate(3, sample(1:1000, 3))
d <- apply(d, 2, function(x) paste(c(x, "\n"), collapse = " "))
d
# [1] "169 807 385 \n" "328 602 604 \n" "125 295 577 \n"


require(stringr)
str_split(d, " ", simplify = T)
# [,1]  [,2]  [,3]  [,4]
# [1,] "169" "807" "385" "\n"
# [2,] "328" "602" "604" "\n"
# [3,] "125" "295" "577" "\n"

Even for large data it is fast:

d <- replicate(1e6, sample(1:1000, 3))
d <- apply(d, 2, function(x) paste(c(x, "\n"), collapse = " "))
d
system.time(s <- str_split(d, " ", simplify = T)) #0.77 sek

Answer 5

Assuming the files are in a CSV, and that all of the 'numbers' are strictly of the form 1 2 or -1 2 (i.e., 1 2 3 or 1 23 are not allowed in the file), then one could start by coding:

# Install package `data.table` if needed
# install.packages('data.table')

# Load `data.table` package
library(data.table)

# Load the CSV, which has just one column named `my_number`.
# Then, coerce `my_number` into character format and remove negative signs.
DT <- fread('file.csv')[, my_number := as.character(abs(my_number))]

# Extract first character, which would be the first desired digit 
# if my assumption about number formats is correct.
DT[, first_column := substr(my_number, 1, 1)]

# The rest of the substring can go into another column.
DT[, second_column := substr(my_number, 2, nchar(my_number))].

Then, if you still really need to create two lists, you could do the following.

# Create the first list.
first_list <- DT[, as.list(first_column)]

# Create the second list.
second_list <- DT[, as.list(second_column)]