There are 2 different ways to implement a solution:
Scan-line
Starting at the coordinate that is at the top (smallest y value), continue to scan down line by line (incrementing y) and see which edges intersect the line.
- For convex polygons you find 2 points, (x1,y) and (x2,y). Simply draw a line between those on each scan-line.
- For concave polygons this can also be a multiple of 2. Simply draw lines between each pair. After one pair, go to the next 2 coordinates. This will create a filled/unfilled/filled/unfilled pattern on that scan line which resolves to the correct overall solution.
In case you have self-intersecting polygons, you would also find coordinates that are equal to some of the polygon points, and you have to filter them out. After that, you should be in one of the cases above.
If you filtered out the polygon points during scan-lining, don't forget to draw them as well.
Flood-fill
The other option is to use flood-filling. It has to perform more work evaluating the border cases at every step per pixel, so this tends to turn out as a slower version. The idea is to pick a seed point within the polygon, and basically recursively extend up/down/left/right pixel by pixel until you hit a border.
Picking a seed point
Usually this is done by randomly selecting a coordinate more or less inside a bound around all corner vertices. No matter which point you select as seed point, first you have to check that the chosen point is either inside, outside or at the very edge of the polygon. You can do this by tracing an imaginary line from the seed point to the 'edge of the canvas' (or infinity). If that imaginary trace intersects with any of the polygon edges a total number of times that is odd, the point lies inside the polygon. If the trace runs exactly through one of the corner vertices, that counts as 2 intersections and won't contribute to 'becoming inside' (for concave polygons - otherwise you can count it as 1). In that case try a different coordinate as seed point. Sometimes it makes sense to evaluate which seed point tends to have a higher fill-ratio based on how far it is from the nearest edge / corner vertex, especially if your fill algorithm is biased to fill faster in a specific direction (e.g. horizontal lines). Usually the approach is to start with points near the center of the bounding box.
Performance
The algorithm has to read and write the entire surface of the polygon, and does not cross self-intersection points. There can be considerable stack-buildup (for naive implementations at least) for large surfaces, and the reduced flexibility you have for the border condition is pixel-based (e.g. flooding into gaps when other things are drawn on top of the polygon). In this sense, this is not a mathematically correct solution, but it works well for many applications.