geom_segment: Why does Colour Work when fixed but not when variable within AES?

Question

I am plotting a circular bar plot using code below (simplified and using iris).

When I state the bars should be white outside AES, it works as expected (see inner plot).

But when I try to make the colour vary based on the data and specifying it within the AES, it either uses the standard colours (see outer colour).

library(tidyverse)
iris2 <- iris %>%
  mutate(id = row_number(),
  colour = if_else(Sepal.Length > 5, "Yellow","White"))
ggplot(iris2) +      
  theme_minimal() +
  theme(
    plot.background = element_rect(fill = "dimgrey"),
    legend.position = "none",
    axis.text = element_blank(),
    axis.title = element_blank(),
    panel.grid = element_blank(),
    plot.margin = unit(rep(0,4), "cm")
  ) +
  coord_polar() +
  ylim(c(0, 20 + max(iris2$Sepal.Length))) +
  # Colour as per fixed colour code (White)
  geom_segment(aes(x = id, xend = id, 
                   y = 10, 
                   yend = 10 + Sepal.Length),
               colour = "White", alpha = 1, 
               size = 1, inherit.aes = FALSE) + 
  # Colour not as per data  
  geom_segment(aes(x = id, xend = id, 
                   y = 20, 
                   yend = 20 + Sepal.Length,
                   colour = colour, alpha = 1),
               size = 1, inherit.aes = FALSE)