One hot encoding creating n-1 dummy variables

Question

In order to one-hot-encode the factor variables in a dataset, I am using the great function of user "Ben" in this post: How to one-hot-encode factor variables with data.table?

one_hot <- function(dt, cols="auto", dropCols=TRUE, dropUnusedLevels=FALSE){
  # One-Hot-Encode unordered factors in a data.table
  # If cols = "auto", each unordered factor column in dt will be encoded. (Or specifcy a vector of column names to encode)
  # If dropCols=TRUE, the original factor columns are dropped
  # If dropUnusedLevels = TRUE, unused factor levels are dropped

  # Automatically get the unordered factor columns
  if(cols[1] == "auto") cols <- colnames(dt)[which(sapply(dt, function(x) is.factor(x) & !is.ordered(x)))]

  # Build tempDT containing and ID column and 'cols' columns
  tempDT <- dt[, cols, with=FALSE]
  tempDT[, ID := .I]
  setcolorder(tempDT, unique(c("ID", colnames(tempDT))))
  for(col in cols) set(tempDT, j=col, value=factor(paste(col, tempDT[[col]], sep="_"), levels=paste(col, levels(tempDT[[col]]), sep="_")))

  # One-hot-encode
  if(dropUnusedLevels == TRUE){
    newCols <- dcast(melt(tempDT, id = 'ID', value.factor = T), ID ~ value, drop = T, fun = length)
  } else{
    newCols <- dcast(melt(tempDT, id = 'ID', value.factor = T), ID ~ value, drop = F, fun = length)
  }

  # Combine binarized columns with the original dataset
  result <- cbind(dt, newCols[, !"ID"])

  # If dropCols = TRUE, remove the original factor columns
  if(dropCols == TRUE){
    result <- result[, !cols, with=FALSE]
  }

  return(result)
}

The function creates n dummy variables for all n factor levels per factor column. But since I want to use the data for modelling, I want only n-1 dummy variables per factor column. Is that possible and if yes, how can I do this using this function?

From my perspective, this line must be adjusted:

newCols <- dcast(melt(tempDT, id = 'ID', value.factor = T), ID ~ value,     drop = T, fun = length)

Here is the input table...

   ID color   size
1:  1 black  large
2:  2 green medium
3:  3   red  small

library(data.table)
DT = setDT(structure(list(ID = 1:3, color = c("black", "green", "red"), 
    size = c("large", "medium", "small")), .Names = c("ID", "color", 
"size"), row.names = c(NA, -3L), class = "data.frame"))

...and the desired output table:

ID color.black color.green size.large size.medium
1 1 0 1 0
2 0 1 0 1
3 0 0 0 0

The `dummyVars` function from package `caret` does exactly this. Is there a reason for not using it? — Jealie, Apr 24 '17 at 16:39

score 3 · Accepted Answer · answered Apr 25 '17 at 16:46

Here goes a solution performing the full-rank dummification (i.e. creating n-1 columns to avoid co-linearity):

require('caret') 
data.table(ID=DT$ID, predict(dummyVars(ID ~ ., DT, fullRank = T),DT))

This does exactly the job:

   ID colorgreen colorred sizemedium sizesmall
1:  1          0        0          0         0
2:  2          1        0          1         0
3:  3          0        1          0         1

See this for a friendly walkthrough of this function, and ?dummyVars for all the available options.

Also: in a comment, the OP mentioned that this operation would need to be done for millions of rows and thousands of columns, thus justifying the need for data.table. If this simple pre-processing step is too much for the "computing muscle", then I am afraid that the modeling step (aka the real deal) is doomed to fail.

One hot encoding creating n-1 dummy variables

1 Answers1