"for" loop only adds the final ggplot layer

Question

Summary: When I use a "for" loop to add layers to a violin plot (in ggplot), the only layer that is added is the one created by the final loop iteration. Yet in explicit code that mimics the code that the loop would produce, all the layers are added.

Details: I am trying to create violin graphs with overlapping layers, to show the extent that estimate distributions do or do not overlap for several survey question responses, stratified by place. I want to be able to include any number of places, so I have one column in by dataframe for each place, and am trying to use a "for" loop to generate one ggplot layer per place. But the loop only adds the layer from the loop's final iteration.

This code illustrates the problem, and some suggested approaches that failed:

library(ggplot2) 

# Create a dataframe with 500 random normal values for responses to 3 survey questions from two cities
topic <- c("Poverty %","Mean Age","% Smokers")
place <- c("Chicago","Miami")
n <- 500
mean <- c(35,  40,58,  50, 25,20)
var  <- c( 7, 1.5, 3, .25, .5, 1)
df <- data.frame( topic=rep(topic,rep(n,length(topic)))
                 ,c(rnorm(n,mean[1],var[1]),rnorm(n,mean[3],var[3]),rnorm(n,mean[5],var[5]))
                 ,c(rnorm(n,mean[2],var[2]),rnorm(n,mean[4],var[4]),rnorm(n,mean[6],var[6]))
                )
names(df)[2:dim(df)[2]] <- place  # Name those last two columns with the corresponding place name.
head(df) 

# This "for" loop seems to only execute the final loop (i.e., where p=3)
g <- ggplot(df, aes(factor(topic), df[,2]))
for (p in 2:dim(df)[2]) {
  g <- g + geom_violin(aes(y = df[,p], colour = place[p-1]), alpha = 0.3)
}
g

# But mimicing what the for loop does in explicit code works fine, resulting in both "place"s being displayed in the graph.
g <- ggplot(df, aes(factor(topic), df[,2]))
g <-   g + geom_violin(aes(y = df[,2], colour = place[2-1]), alpha = 0.3)
g <-   g + geom_violin(aes(y = df[,3], colour = place[3-1]), alpha = 0.3)
g

## per http://stackoverflow.com/questions/18444620/set-layers-in-ggplot2-via-loop , I tried 
g <- ggplot(df, aes(factor(topic), df[,2]))
for (p in 2:dim(df)[2]) {
  df1 <- df[,c(1,p)]
  g <- g + geom_violin(aes(y = df1[,2], colour = place[p-1]), alpha = 0.3)
}
g
# but got the same undesired result

# per http://stackoverflow.com/questions/15987367/how-to-add-layers-in-ggplot-using-a-for-loop , I tried
g <- ggplot(df, aes(factor(topic), df[,2]))
for (p in names(df)[-1]) {
  cat(p,"\n")
  g <- g + geom_violin(aes_string(y = p, colour = p), alpha = 0.3)  # produced this error: Error in unit(tic_pos.c, "mm") : 'x' and 'units' must have length > 0
  # g <- g + geom_violin(aes_string(y = p            ), alpha = 0.3)  # produced this error: Error: stat_ydensity requires the following missing aesthetics: y
}
g
# but that failed to produce any graphic, per the errors noted in the "for" loop above

Answer 1

The reason this is happening is due to ggplot's "lazy evaluation". This is a common problem when ggplot is used this way (making the layers separately in a loop, rather than having ggplot to it for you, as in @hrbrmstr's solution).

ggplot stores the arguments to aes(...) as expressions, and only evaluates them when the plot is rendered. So, in your loops, something like

aes(y = df[,p], colour = place[p-1])

gets stored as is, and evaluated when you render the plot, after the loop completes. At this point, p=3 so all the plots are rendered with p=3.

So the "right" way to do this is to use melt(...) in the reshape2 package so convert your data from wide to long format, and let ggplot manage the layers for you. I put "right" in quotes because in this particular case there is a subtlety. When calculating the distributions for the violins using the melted data frame, ggplot uses the grand total (for both Chicago and Miami) as the scale. If you want violins based on frequency scaled individually, you need to use loops (sadly).

The way around the lazy evaluation problem is to put any reference to the loop index in the data=... definition. This is not stored as an expression, the actual data is stored in the plot definition. So you could do this:

g <- ggplot(df,aes(x=topic))
for (p in 2:length(df)) {
  gg.data <- data.frame(topic=df$topic,value=df[,p],city=names(df)[p])
  g <- g + geom_violin(data=gg.data,aes(y=value, color=city))
}
g

which gives the same result as yours. Note that the index p does not show up in aes(...).

---

Update: A note about scale="width" (mentioned in a comment). This causes all the violins to have the same width (see below), which is not the same scaling as in OP's original code. IMO this is not a great way to visualize the data, as it suggests there is much more data in the Chicago group.

ggplot(gg) +geom_violin(aes(x=topic,y=value,color=variable),
                        alpha=0.3,position="identity",scale="width")

Answer 2

You can do it w/o a loop:

df.2 <- melt(df)
gg <- ggplot(df.2, aes(x=topic, y=value))
gg <- gg + geom_violin(position="identity", aes(color=variable), alpha=0.3)
gg

Answer 3

You can use aes_() rather than aes(), which appears to stop the lazy evaluation. Answer found on a closed question that links here (Update a ggplot using a for loop (R)), but thought it should be here since the other question was closed.

Answer 4

While generally speaking, reshaping the data is always preferred, with newer version of ggplot2 (>3.0.0), you can use !! to inject values into the aes() For example you can do

g <- ggplot(df, aes(factor(topic), df[,2]))
for (p in 2:dim(df)[2]) {
  g <- g + geom_violin(aes(y = df[,!!p], colour = place[!!p-1]), alpha = 0.3)
}
g

To get the desired result. The !! will force evaluation rather than remaining lazy as is the default.