Let me try to tackle these one by one...
Q: What are the *proj files for?
The *proj files are the native language of MSBuild (or xbuild if you're on Mono). In the simplest case, they just list all the files to be compiled and all references to other projects that they use. And a few other properties, like target platform, CPU architecture, etc. The original idea was that one *proj file produces one compilation assembly unit (aka "DLL", which roughly corresponds to JAR). This is still mostly true, but not always. For example, TypeScript projects can produce multiple JS files.
But the core architecture of MSBuild makes these files very flexible. They are basically built on a system of plugins (called "Tasks"), where each Task does some specific thing. A number of tasks come in the box, such as "compile C# files" or "output to log", but you can also add your own, or install some in the form of packages, or globally. Plus, the core of MSBuild allows for some tricks, like variables (kind of), loops (sort of), and branching (to an approximation), to the point where you can actually write kind of real programs entirely in a *proj file. The original idea behind all this was that MSBuild would become the primary vehicle for the whole build process, no other tools required. And for simple toy projects it kind of is: when all you need is just compile a bunch of files to a DLL, MSBuild does the job marvelously. And even with not-so-simple projects, people have tried to do the whole build entirely in *proj files. There are some projects that do this even now.
However, over time it became clear that writing high-level build logic in XML is so tricky and backwards that it becomes unmaintainable very fast. And so a bunch of tools appeared specifically for implementing the high-level build logic. Which brings me to...
Q: Isn't FAKE a replacement for MSBuild?
Yes, it is. Well, no, actually it isn't. Well, maybe. Kind of.
As I said above, it is possible to write the whole build logic in MSBuild: compile, copy, compress (where needed), bundle (if required), package (where appropriate), and publish (where legal :-). But it is extremely awkward to write and quickly becomes unmaintainable.
Enter FAKE: in FAKE, you write the whole logic of the build process entirely in F#. This means you can use libraries, abstractions, higher-order functions, special types - all the goodness of a real high-level language. You can specify the list of source files with glob patterns, call the F# compiler with FscHelper, run tests with XUnitHelper, package and publish with Paket helper, deploy to Azure, and even notify your teammates on Slack - all without leaving the comfort of the functional goodness.
There is one problem though: the build script is not analyzable by tools.
Since it's a real program, and not a data format, an IDE wouldn't know how to tease the list of source files out of it, or how to add new ones; package manager wouldn't know where to insert references to packages, and so on.
The MSBuild files, on the other hand, are the definition of analyzability: they're XML, and there are strong standards in place about what goes where.
And so it became: we use MSBuild to specify some strict, yet simple properties, and then use a FAKE script to "drive" the higher-level build logic - such as running tests, generating docs, publishing, deploying, etc. And instead of specifying a list of files and calling F# compiler directly, our build script just calls MSBuild.
Q: Why does Paket need both references and dependencies?
In short: because we usually deal with more than one compilation unit (aka "Project") at once (we call such system of projects a "Solution"), and we don't like it when different projects under one solution use different versions of the same packages: conflicts abound, and our heads explode. So we specify one list of packages for the whole solution (in "dependencies"), and then every project can pick the ones that it actually uses (in "references").
In long: @rmunn's excellent answer discusses this in a bit more detail.
Also of interest: the "native" .NET package manager NuGet doesn't actually have this property. Every project under NuGet has its own, completely independent list of packages. And this head-exploding version conflicts do actually happen. On large solutions, depressingly often. This is one of the many reasons why Paket is superior.
