Using colors from column in ggplot2

Question

I have a tibble that looks like this

utime_start | utime_end  |  color
------------------------------
1486123312  | 1486123825 | 'ff00ff'
1486123826  | 1486123829 | 'ff0aff'

I want to use to visualize it using the colors specified in the color column. Something like this:

(data
  %>% mutate(start = as_datetime(utime_start))
  %>% mutate(end = as_datetime(utime_end))
  %>% ggplot()
  + geom_segment( aes(x = start, y = 0, xend = end, yend=1, color = color) )
  + xlab('Date')
)

How do I tell ggplot to use colors from a column, rather then its own colorscale?