Dealing with non-convex polygons will be tricky, so perhaps you should start off by dividing your polygon into multiple convex polygons. You can use the interior angle to decide where a corner is non-convex, and apply cuts that involve these corners at the minimum. But perhaps matlab does support some triangulation algorithm you can use.
Once you have your convex polygons, you can think of them as intersections of half spaces. If A and B are two corners which span a line, and P is a grid point, you can use the sign of the determinant
| Ax Bx Px |
| Ay By Py |
| 1 1 1 |
to decide on which side of the line the point lies. Which sign is which depends on the order in which you walk around your polygon, but if you process corners in order, then a point P lies within the convex polygon iff the sign never changes. Horizontal or vertical lines won't be special cases in this formulation, and you won't have divisions either which is good for performance and might help with exactness as well.
If you don't want to iterate over all grid points using that method, you can come up with various optimizations. One would be to precompute the cross product A × B of the two corners spanning each line. The dot product between that cross product and a point P is equal to the above determinant (i.e. det(A,B,P)=(A×B)·P), so instead of the full determinant you now only have to compute three products and two sums for each point-line combination.
If you want to optimize this even further, you'd probably best look at something like Bresenham's line algorithm to compute the points at the border of the polygon, and then simply enumerate all points on a horizontal (or vertical) line between the matching border points.
Unless you perform all your computations with integers or other exact numbers, rounding issues might be a major concern in all of this. You have to decide whether or not to count border points, and also make sure to count points inside the original polygon but on the border of its convex portions exactly once. How much effort this requires depends to a large degree on how your input looks like.