If you really don't want to use OpenCV or another Computer Vision Library you can implement the algorithms necessary.
I'm not sure why your binary file would be super large. You'd only need to include the code for imgproc and core (I think, it's been a little while since I've used it).
You'll need to segment your image. Your foreground being the shapes you want to test and the background everything else. This isn't too bad if you know what color they are or what color the background is).
Compute the connected components in your foreground. This will extract pixels forming a blob. (a simple linear pass going from top left to bottom right and bottom right to top left will do this in 2N time).
Once you have the connected components you can compute the 0th 1st and 2nd central image moments for each. https://en.wikipedia.org/wiki/Image_moment is pretty good.
The 0th moment is the area of connected component. First moments are the centroid of the blob. The second moments can tell you the rotation of the object. The Eigen values of the 2nd central moments covariance matrix will be proportional to the major and minor axis of your rectangle.
This will give you the information about your blob. I.E. its location the image and it's rotation.
To determine if that blob is a rectangle there are several methods.
You can use a Hough Transform see Rectangle detection with Hough transform
Or you can try to compute the expected compactness (Perimeter / area) of a rectangle with the Given Area and major (length) and minor (width) axis proportionality to the actual compactness of the blob. If it's close enough then you have a rectangle like object. (depending on how close is close enough you may get things that are more oval than rectangular).
While these algorithms aren't particularly hard to implement it may be difficult to get all of them working perfectly together. It may be easier for you to find what parts of OpenCV you need to include in your binary and only include those portions.