5

The problem that I have with make always rebuilding Makefile targets (make always rebuilds Makefile targets) and its investigation uncovered another issue, which is the subject of this question. Repeated execution of the following R code results in a loss of objects' attributes during data transformation operations.

For the record, I have to say that I've already written on this subject (Approaches to preserving object's attributes during extract/replace operations), but that question and answer were more general (and I was incorrect that simple saving attributes works - it worked for me as of that writing, because at the time I haven't been performing operations, potentially dangerous for objects' attributes).

The following are excerpts from my R code, where I'm experiencing loss of attributes.

##### GENERIC TRANSFORMATION FUNCTION #####

transformResult <- function (dataSource, indicator, handler) {

  fileDigest <- base64(indicator)
  rdataFile <- paste0(CACHE_DIR, "/", dataSource, "/",
                      fileDigest, RDS_EXT)
  if (file.exists(rdataFile)) {
    data <- readRDS(rdataFile)

    # Preserve user-defined attributes for data frame's columns
    # via defining new class 'avector' (see code below)). Also,
    # preserve attributes (comments) for the data frame itself.
    data2 <- data.frame(lapply(data, function(x) 
      { structure(x, class = c("avector", class(x))) } ))
    #mostattributes(data2) <- attributes(data)
    attributes(data2) <- attributes(data)

    result <- do.call(handler, list(indicator, data2))
    saveRDS(result, rdataFile)
    rm(result)
  }
  else {
    error("RDS file for \'", indicator, "\' not found! Run 'make' first.")
  }
}


## Preserve object's special attributes:
## use a class with a "as.data.frame" and "[" method

as.data.frame.avector <- as.data.frame.vector

`[.avector` <- function (x, ...) {
  #attr <- attributes(x)
  r <- NextMethod("[")
  mostattributes(r) <- attributes(x)
  #attributes(r) <- attr
  return (r)
}

##### HANDLER FUNCTION DEFINITIONS #####

projectAge <- function (indicator, data) {

  # do not process, if target column already exists
  if ("Project Age" %in% names(data)) {
    message("Project Age: ", appendLF = FALSE)
    message("Not processing - Transformation already performed!\n")
    return (invisible())
  }

  transformColumn <- as.numeric(unlist(data["Registration Time"]))
  regTime <- as.POSIXct(transformColumn, origin="1970-01-01")
  prjAge <- difftime(Sys.Date(), as.Date(regTime), units = "weeks")
  data[["Project Age"]] <- as.numeric(round(prjAge)) / 4 # in months

  # now we can delete the source column
  if ("Registration Time" %in% names(data))
    data <- data[setdiff(names(data), "Registration Time")]

    if (DEBUG2) {print(summary(data)); print("")}

  return (data)
}


projectLicense <- function (indicator, data) {

  # do not process, if target column (type) already exists
  if (is.factor(data[["Project License"]])) {
    message("Project License: ", appendLF = FALSE)
    message("Not processing - Transformation already performed!\n")
    return (invisible())
  }

  data[["Project License"]] <- 
    factor(data[["Project License"]],
           levels = c('gpl', 'lgpl', 'bsd', 'other',
                      'artistic', 'public', '(Other)'),
           labels = c('GPL', 'LGPL', 'BSD', 'Other',
                      'Artistic', 'Public', 'Unknown'))

  if (DEBUG2) {print(summary(data)); print("")}

  return (data)
}


devTeamSize <- function (indicator, data) {

  var <- data[["Development Team Size"]]

  # convert data type from 'character' to 'numeric' 
  if (!is.numeric(var)) {
    data[["Development Team Size"]] <- as.numeric(var)
  }

  if (DEBUG2) {print(summary(data)); print("")}

  return (data)
}


##### MAIN #####

# construct list of indicators & corresponding transform. functions
indicators <- c("prjAge", "prjLicense", "devTeamSize")
transforms <- list(projectAge, projectLicense, devTeamSize)

# sequentially call all previously defined transformation functions
lapply(seq_along(indicators),
       function(i) {
         transformResult("SourceForge",
                         indicators[[i]], transforms[[i]])
         })

After the second run of this code, names "Project Age" and "Project License" as well as other user-defined attributes of the data frame data2 are lost.

My question here is multifaceted:

1) what statements in my code could lead to loss of attributes AND WHY;

2) what is the correct line of code (mostattributes <- attributes or attributes <- attributes/attr) in transformResult() and avector class definition AND WHY;

3) is the statement as.data.frame.avector <- as.data.frame.vector really needed, if I add class attribute avector to a data frame object and, in general, prefer a generic solution (applicable not only to data frames); WHY OR WHY NOT.

4) saving via attr in class definition doesn't work, it fails with the following error:

Error in attributes(r) <- attr :
  'names' attribute [5] must be the same length as the vector [3]
Calls: lapply ... summary.data.frame -> lapply -> FUN -> summary.default -> [ -> [.avector

So, I had to go back to using mostattributes(). Is it OK?

==========

I have read the following on the subject:

  1. SO question: How to delete a row from a data.frame without losing the attributes (I like the solution by Ben Barns, but it differs a bit from the one suggested by Gabor Grothendieck and Marc Schwartz - see below);

  2. SO question: indexing operation removes attributes (while the solution is legible, I prefer one, based on class definition /sub-classing?/);

  3. A generic solution suggested by Heinz Tuechler (https://stat.ethz.ch/pipermail/r-help/2006-July/109148.html) - Do I need this?;

  4. An explanation by Brian Ripley (http://r.789695.n4.nabble.com/Losing-attributes-in-data-frame-PR-10873-tp919265p919266.html) - I found it somewhat confusing;

  5. A solution suggested by Gabor Grothendieck (https://stat.ethz.ch/pipermail/r-help/2006-May/106308.html);

  6. An explanation of Gabor Grothendieck's solution by Marc Schwartz (https://stat.ethz.ch/pipermail/r-help/2006-May/106351.html) - very nice explanation;

  7. Sections 8.1.28 and 8.1.29 of the "R Inferno" book (www.burns-stat.com/pages/Tutor/R_inferno.pdf) - I've tried his suggestions of using storage.mode(), but doesn't really solve the problem, as coercing via storage doesn't affect class of an object (not to mention that it doesn't cover other than coercion attribute-clearing operations, such as subsetting and indexing;

  8. http://adv-r.had.co.nz/Data-structures.html#attributes;

  9. http://stat.ethz.ch/R-manual/R-devel/library/base/html/attributes.html;

  10. http://cran.r-project.org/doc/manuals/r-devel/R-lang.html#Copying-of-attributes.

P.S. I believe that this question is of general nature, so I haven't provided a reproducible example at this time. I hope that it's possible to answer this without such example, but, if not, please let me know.

Community
  • 1
  • 1
Aleksandr Blekh
  • 2,462
  • 4
  • 32
  • 64
  • 1
    There is no problem ever that has not benefitted from a reproducible example. It's exponentially more difficult to debug code by reading it than it is by running it. If the above code was meant to be runnable, I get the error: `could not find function "base64"`. This problem doesn't look simplified or isolated and you seemed to have combined multiple questions into one post. You've made this a very, very difficult question to answer. I would recommend editing it. – MrFlick Jun 03 '14 at 02:05
  • @MrFlick: Sorry about the trouble and thank you for trying to help! The code I provided was NOT meant to be runnable, as I thought sometimes it's enough to look at code rather than run it. So, I intentionally minimized the code to show essential parts only. As for `base64()`, it a function from `RCurl`. I will consider editing the question, but I still believe that most, if not all, of my questions can be answered by reviewing code. The problem with reproducible example (RE) is that sometimes an issue someone is trying to solve lies in a specific context, so RE isn't useful for that. – Aleksandr Blekh Jun 03 '14 at 03:12
  • 1
    That's a lot to ask someone to read line by line when we don't really even know what the exact error is. If you could narrow it down to just one function, that may be helpful. Problems are impossible to solve in the general sense (ie "how do i make my program always work"). Reproducible examples isolate one particular problem that is testable and verifiable. – MrFlick Jun 03 '14 at 03:19
  • @MrFlick: I agree with you in general. And I understand the value of REs. However, the problem with REs is that they not only isolate problems, as you mentioned, but, at the same time, they remove the context. Interaction with other functions and parts of code (context) often represents the source of problems people are trying to solve. So, asking someone to ALWAYS use REs might not be always beneficial for helping solving those problems. Having said that, I'll do my best to isolate the functionality and update my questions. – Aleksandr Blekh Jun 03 '14 at 04:08

1 Answers1

0

I'm answering my own question - well, for now, only partially:

1) Under more intense investigation and after some code updates, it appears that attributes in fact are NOT being lost (still trying to figure out what changes caused the expected behavior - will report later).

2) I have figured out the reason of intermittent output and losing all cache data after the transformation, as follows. During multiple subsequent runs of the code, the second run of each transformation (handler) function (projectAge(), projectLicense() and devTeamSize()) returns NULL, since the transformation has already been done:

if (<condition>) {
  ...
  message("Not processing - Transformation already performed!\n")
  return (invisible()) # <= returns NULL
}

The returned NULL then was getting passed to saveRDS(), thus, causing the loss of cache data.

I fixed this problem by simple validation of result before saving the transformed object:

# the next line is problematic due to wrong assumption of always having full data returned
result <- do.call(handler, list(indicator, data2))
if (!is.null(result)) saveRDS(result, rdataFile) # <= fixed by validating incoming data

That's it so far, thanks for reading! I will be updating this answer until all the issues are clarified.

Aleksandr Blekh
  • 2,462
  • 4
  • 32
  • 64