Int32.MaxValue (using the value given here) is 2,147,483,647.
In base 2, that is: 111 1111 1111 1111 1111 1111 1111 1111
... 2^31-1. The first bit is the sign bit.
If you multiply that by itself, you get: 11 1111 1111 1111 1111 1111 1111 1111 0000 0000 0000 0000 0000 0000 0000 0001
Going back to the original problem of "why is it 1?", since Integer.MaxValue is the maximum value, that causes an integer overflow. The result is truncated to the lowest 31 bits, which is all 0s plus 1.
Edit: Here is a tutorial on binary multiplication. Using a simple case of all 1s:
111
* 111
you get:
00111
01110
+ 11100
= 100001
You can expand this for the case of Int32.MaxValue. I shortened it to 3 digits for brevity.
Also, as another answer said, in C# these overflows will occur by default.