-allow forks
-allow only private forks
-no forks
These three options are only available for private repos.
Allow forks means it both support public forks (the fork version can be seen publicly) and private forks (the fork version can only seen by the person who forked it).
Allow only private forks means it allow private-only forks.
No forks means you repo don't allow any fork.
And fundamentally, to fork a project (take the source from someone's repository at certain point in time, and apply your own diverging changes to it), you would clone the remote repository to create a copy of it, then do your own work in your local repository and commit changes.
For differences between git and mercurial type of repositories, you can check out this post: What is the Difference Between Mercurial and Git?