I think you do not need to use fill
- instead include density
and angle
as below. Would set the legend to correspond to levels
in your bosnia$group
factor. In addition, increased density to 10, 15, and 20 and increased size (cex
) for clarity.
par(mar=c(1.1, 9.1, 1.1, 1.1), xpd=TRUE)
plot(bosnia,density=c(10,15,20)[bosnia$group], angle=c(0,45,90)[bosnia$group])
legend("topleft", inset=c(-0.4,0), ncol = 1,
legend = levels(bosnia$group),
density = c(10, 15, 20),
angle = c(0, 45, 90),
cex = 1.5,
bg = "white")

Update: If you wish to increase just the size of the boxes in the legend, but not the text, you will need to create a revised legend
function.
As mentioned in the comment, copy/paste the legend
function (type legend
in the console, copy and paste to editor), and rename new function something like legend_large_box
:
legend_large_box <- function (x, y = NULL, legend, fill = NULL, col = par("col"),
border = "black", lty, lwd, pch, angle = 45, density = NULL,
bty = "o", bg = par("bg"), box.lwd = par("lwd"),
box.lty = par("lty"), box.col = par("fg"), pt.bg = NA,
cex = 1, pt.cex = cex, pt.lwd = lwd, xjust = 0, yjust = 1,
x.intersp = 1, y.intersp = 1, adj = c(0, 0.5), text.width = NULL,
text.col = par("col"), text.font = NULL, merge = do.lines &&
has.pch, trace = FALSE, plot = TRUE, ncol = 1, horiz = FALSE,
title = NULL, inset = 0, xpd, title.col = text.col, title.adj = 0.5,
seg.len = 2)
{...
In your example, fill is not included in the statement, so the lines you need to change for the larger box in this particular plot are:
xbox <- xc * 0.8
ybox <- yc * 0.5
Try something like 1.2 and 0.8 instead of 0.8 and 0.5.
Then call legend_large_box
:
legend_large_box("topleft", inset=c(-0.4,0), ncol = 1,
legend = levels(bosnia$group),
density = c(10, 15, 20),
angle = c(0, 45, 90),
bg = "white")