None of the reasons given by the answer adequately addresses the question posed, which is "Why is the Git index file binary?". The accepted answer is just not correct. The index doesn't "contain" any plain-text files--it contains references to plain-text files. Furthermore, to say that the Git index contains "index entries" says really nothing useful at all, especially to a fellow developer seeking Truth... Finally, trees are not cached by the index--references to the trees are cached.
The index isn't binary because it's "indexed" (as the poster concluded in a comment above)--and it isn't binary for "performance reasons", per se. Everything in the index could be expressed using a pure text file--even the flags and bits expressed within the binary index file could be expressed as ASCII. It's binary because binary file formats that contain bit-wise flags are able to use disk space more efficiently. And, knowing Linus, it probably is binary so as to dissuade tampering by newbies with easy-access to text editors.
* New information *
Version 4 of the index implements path compression, saving up to roughly 50% on the size of the index for large repos. (Source: https://git-scm.com/docs/git-update-index) This compression would lend itself to a binary-format index file.