How to map specific colours to geom_tile heatmap in R?

Question

Similar questions about heatmap colour scales have been asked here and here. However, they don't really answer my question.

I am trying to map specific colour values to specific tiles using geom_tile for a heatmap. For example, if I create a basic heatmap, like so:

library(ggplot2)
x <- LETTERS[1:3]

data <- expand.grid(var1 = x, var2 = x)
data$value <- runif(9, 0, 5)

pal <- rev(viridis::viridis(4))

# Heatmap 
p <- ggplot(data, aes(var1, var2)) + 
  geom_tile(aes(fill= data$value)) +
  scale_fill_gradientn(
    colors = pal,
    limits = c(0,5),
    name = "value") +
  theme(aspect.ratio = 1)
p

This creates something like this:

However, my data has a column of specific colours associated with each row. Something like this:

data$newCols <- c(
  "#440154",
  "#88B2D9",
  "#ff0000",
  "#FFED78",
  "#D8B7E7",
  "#B4DAFF",
  "#56CE90",
  "#B4DAFF",
  "#6B477A"
)

So now my data will look something like this:

> data
  var1 var2     value newCols
1    A    A 1.8215538 #440154
2    B    A 1.4818806 #88B2D9
3    C    A 4.7236024 #ff0000
4    A    B 2.8583989 #FFED78
5    B    B 3.6576457 #D8B7E7
6    C    B 0.0307718 #B4DAFF
7    A    C 4.6653170 #56CE90
8    B    C 3.2858039 #B4DAFF
9    C    C 4.0803276 #6B477A

I want to create a new heatmap, but this time I want the colour of the tile associated with each pair of variables var1 and var2 to be the corresponding colour in that row. So, for example, the colour in tile A-A will be #440154... the colour in tile B-A will be #88B2D9... etc.

Any suggestions as to how I can use the newCols column as the colours in my heatmap.

I naively tried to do:

p <- ggplot(data, aes(var1, var2)) + 
  geom_tile(aes(fill= data$value)) +
  scale_fill_gradientn(
    colors = newCols,
    limits = c(0,5),
    name = "value") +
  theme(aspect.ratio = 1)

But it's not placing the colours in the correct tiles