TBH, I'm unsure what you're after: whether this is about how to plot a heatmap? Or how to calculate a cosine dissimilarity matrix "by hand"? Either way, here is a worked-through example using some sample data.
For future postings, please learn how to ask good questions, and always provide a minimal reproducible example/attempt including sample data.
# Function to calculate cosine dissimilarity matrix
# Thanks to: https://stats.stackexchange.com/a/149865/121489
cos.sim <- function(ma, mb) {
mat <- tcrossprod(ma, mb);
t1 <- sqrt(apply(ma, 1, crossprod));
t2 <- sqrt(apply(mb, 1, crossprod));
return(mat / outer(t1, t2));
}
# Generate sample data
set.seed(2017);
x1 <- matrix(rnorm(45), ncol = 5);
# Calculate the cosine dissimilarity matrix
m <- cos.sim(x1, x1);
# Show heatmap
library(tidyverse);
m %>%
as_tibble() %>%
rownames_to_column("x1") %>%
gather(x2, value, 2:10) %>%
mutate(x2 = gsub("V", "", x2)) %>%
ggplot(aes(x1, x2)) + geom_tile(aes(fill = value));

Note the 1 diagonal of m
in the heatmap.