Axis Labels that are ggplot2 objects / grobs

Question

I wish to use ggplot2 objects/grobs/plots as axis labels.

Here is my toy example:

library(dplyr)
library(ggplot2)
# master plot
df <- data_frame(y = c("unchanging", "increasing", "decreasing"), x = c(20, 50, 30))
ggplot(df, aes(x, y)) + geom_point()

# fxn generates ggplot2 object specifying a line plot from two points
two_pt_line_plot <- function(y1, y2) {
  df <- data_frame(y = c(y1, y2), x = c("from", "to"))
  ggplot(df, aes(x,y, group = 1)) + geom_line(size = 4) + 
    xlab(NULL) + ylab(NULL) + 
    scale_x_discrete(expand=c(0,0)) + 
    scale_y_continuous(expand=c(0,0))
}
# make the three plot objects, name them appropriately.
grobs <- Map(two_pt_line_plot, c(.5,0,1), c(.5, 1, 0))
names(grobs) <- df$y
grobs
#> $unchanging

#> $increasing

#> $decreasing

I want to programmatically generate this:

The only thing I can currently think of is that I somehow layer over the plots as facets where the theming as been hacked to the max to make it look like it belongs. But I haven't been able to do that yet and it seems like a very hack-y solution. I therefore thought I would throw it out there.