Version control should mainly keep "primary objects". Primary objects are files that are not derived automatically from other files. If some tool generates B from A, then only A should be in version control, at least ideally. Some circumstances can justify B being in version control also. For instance, the program has to build in environments where the A to B tool doesn't exist.
An example occurs in compiler bootstrapping. Suppose that the project implements a language called L, whose compiler outputs C. Most of L is implemented in L itself! Oops, many of the target users do not have an L compiler to build the L sources; they only have a C compiler. Those users cannot pull the repo and build the L compiler unless the repository includes the C versions of the L source files (or they otherwise obtain them somehow).
Large, binary files can be primary objects. For instance, the image data for a video game and such. There is definitely a need for version control that handles large, binary files.
One way to handle large binaries in a version control system that do not work with such files very well is to keep the binaries on some server (under versioned paths), and store just those paths in the repo (in some parametrized way so that if the paths have to change, the users of the repo just change some environment variable).
Sometimes binaries are the derived objects of some other repo. For instance, you have some embedded system project that has all sorts of software in the repo. One of the pieces is some piece of firmware that is uploaded to some chip when the system boots. That comes from some other repo; you don't build it. So just the binary images of that firmware are checked in. The firmware is a derived object from some primaries, but you either don't have them, or don't want to pull in those primaries because of dependencies (like the whole toolchain needed to build them and such).