0

I have a large dataframe containing data from several participants. Each participant's data extends over many rows and I need to save the data for each participant in a separate text file. As suggested elsewhere (Split dataframe into multiple output files), I am using data.table::fwrite for this job. However, this prompts a message and I am unsure about its interpretation in my case:

Empty data.table (0 rows) of 1 col: A

I have already searched for possible explanations, but felt that none of these directly addressed my problem (e.g. see: unclear error/message: "Empty data.table (0 rows) of 1 col:"; https://github.com/Rdatatable/data.table/issues/3262).

To illustrate, I have generated a sample script, which is similar in nature and replicates this message:

# generate data table
DT = data.table(A=c(1,2,1,2,1,2), B=c(1,2,1,1,2,2), C=c(1:6), D=c(20:15))

   A B C  D
1: 1 1 1 20
2: 2 2 2 19
3: 1 1 3 18
4: 2 1 4 17
5: 1 2 5 16
6: 2 2 6 15

# save data in separate files based on column A
DT[, fwrite(.SD, stringr::str_c("T", unique(A),".csv")), by=.(A)]

This generates the above message. Nonetheless, two files are created (as needed) and contain the relevant data. 

fread("T1.csv")

  B C  D
1: 1 1 20
2: 1 3 18
3: 2 5 16

fread("T2.csv")

   B C  D
1: 2 2 19
2: 1 4 17
3: 2 6 15

The message prompt suggests to me, however, that something in my script didn't quite work (I don't intend to have, or save, empty data tables). Is something amiss here indeed or is it just the message prompt that's erroneous? Interestingly, when removing fwrite, the message disappears, suggesting to me that the message prompt is linked to fwrite:

 DT[, .SD, by=.(A)]

1: 1 1 1 20
2: 1 1 3 18
3: 1 2 5 16
4: 2 2 2 19
5: 2 1 4 17
6: 2 2 6 15

Any advice (and explanations) would be much appreciated.

Tiberius
  • 331
  • 1
  • 9
  • 1
    the msg is returned by the output of the `j` argument in `DT[,....]` call. since `fwrite` does not return anything, the `DT[,...]` returns empty `data.table`. Nothing to worry about. – chinsoon12 Jan 24 '19 at 00:35
  • @chinsoon12 If that is the case then can you please explain why fn <- function(x)return(str(x)); DT[,fn(.SD), by=.(A)] gives 'Empty data.table (0 rows) of 1 col: A' at the end. – Raja Saha Jan 24 '19 at 10:00
  • @RajaSaha check out the Value section in `?str` – chinsoon12 Jan 24 '19 at 10:02
  • @chinsoon12 Got it.. Thanks.. – Raja Saha Jan 24 '19 at 10:29
  • @chinsoon12: Thanks, I see. In other words, datatable considers the output of `j` and `fwrite` as two separate things. Whilst in my case `fwrite` has a legitimate output (i.e., the .csv files), it does not pass on any additional information to `j` that `j` could use to generate output. So, the message just says that `j` has no output, but this makes no reference to any potential output that `fwrite` may have. That is, the message makes no statement on the content or quality of the text files, i.e., I can trust that the text files are fine. – Tiberius Jan 24 '19 at 15:33

0 Answers0