Welcome to SO.
Matplotlib still doesn't have an easy way to map integers to colors.
Usually, the most straightforward way is to simply apply the mapping outside of matplotlib and then pass the color values to matplotlib.
import numpy as np
import matplotlib.pyplot as plt
n = 10
x = np.random.rand(n)
y = np.random.rand(n)
color_as_integer = np.random.randint(3, size=n)
colormap = {
0 : np.array([ 0, 0, 0, 255]), # unlabelled
1 : np.array([ 70, 70, 70, 255]), # building
2 : np.array([100, 40, 40, 255]), # fence
}
# matplotlib works with rbga values in the range 0-1
colormap = {k : v / 255. for k, v in colormap.items()}
color_as_rgb = np.array([colormap[ii] for ii in color_as_integer])
plt.scatter(x, y, s=100, c=color_as_rgb)
plt.show()
You can then use proxy artists to create a legend as outlined here.
The alternative is to use a combination of ListedColormap and BoundaryNorm to map integers to colors, as outlined in this answer.
In that case, you can also get a colorbar as outlined here (although making a proper legend is probably better in your case).