Smoothing and continuous color gradient in ggplot2

Question

I am making plots using ggplot2 in R, and I have trouble combining smoothing and a continuous color scale. More specifically, I would like to draw a bunch of smoothed lines and for each of them, I would like to have their coloring change over the x-axis, such that they are e.g. darkest near their right endpoints. If I were to do this with pointwise linear curves (instead of smoothed lines), I would do something like

d <- data.frame(id = rep(1:100, 10), x = rep(1:10, each = 100), 
                y = rep(1:10, each = 100) + rnorm(1000),
                z = factor(rep(rep(c("a", "b"), each = 10), 500)))


ggplot(d, aes(x = x, y = y, group = id, col = x)) +
       geom_line()  

which works perfectly fine. However, if I try using a smoother rather than just connecting points, I do not get the same result: All lines simply become black using the following code:

ggplot(d, aes(x = x, y = y, group = id, col = x)) +
       geom_line(stat = "smooth", method = "loess")

Any hints as to why this happens and what can be done about it would be very much appreciated! I have seen this post, which suggests that one has to smooth the data before plotting, but I would very much like to do everything in my ggplot() call.

---

I have tried two things worth mentioning already. First, using the geom_smooth() function directly does not make a difference (but does change the default line-color):

ggplot(d, aes(x = x, y = y, group = id, col = x)) +
       geom_smooth(se = FALSE, method = "loess")

Secondly, col does seem to be the correct parameter to target, since when coloring is chosen according to a discrete variable, everything works:

ggplot(d, aes(x = x, y = y, group = id, col = z)) +
       geom_line(stat = "smooth", method = "loess")  

Answer 1

My suggestion in the other question is still the "right" way to do it. If you really don't want to modify your original dataframe, you can pipe your way through the broom package, with something like:

d %>% 
 group_by(id) %>% 
 do(augment(loess(y~x, data = .))) %>% 
 ggplot(aes(x = x, y = .fitted, group = id, colour = x)) +
 geom_line(stat = "identity", aes(colour = x))

Throughout I'm using only a subset of the data (d %>% filter(id %in% 1:10)) to make it clearer/faster:

While this way is more "elegant", it means that you have to run the model fit every time you re-draw the figure (which also happens when you use stat_smooth() by the way). This can make performance (very) slow.

In addition, you'll notice the lines are kinky, not smooth. They're smoothed from the raw data, but the gap between each x value is too large to produce an indistinguishable curve.

The way around this is to make explicit what stat_smooth is doing: calculating a new dataframe of xs and ys from the model. To do that, you supply newdata= to augment. The side effect of this is you lose your old y (and z) values.

d %>% 
 group_by(id) %>% 
 do(augment(loess(y~x, data = .),
      newdata = data.frame(x = 0.1*(1:100)))) %>% 
 ggplot(aes(x = x, y = .fitted, group = id, colour = x)) +
 geom_line(stat = "identity", aes(colour = x))

The most hackish and inadvisable method is to use stat_smooth's internally calculated variables, which are mostly undocumented and subject to change without notice. Hadley Wickham explicitly discourages this.

But let's throw caution to the wind!

d %>% 
  ggplot(aes(x = x, y = y, group = id, colour = x)) +
  geom_line(stat = "smooth", method = "loess", aes(colour = ..x..))

Finally, of course you can put any sort of algebraic expression in for colour=. Try colour = sin(x^2/2).

This illustrates why this hasn't been coded in as an intentional use case. It's ugly, doesn't add information, and distracts from the actual information. So maybe stop and think long and hard about why it is you want to do this at all.