one of the developers from OpenProject here.
Short answer: There is no current built-in functionality for Mercurial (only Subversion and Git created through OpenProject), but adding your own as a plugin is not that difficult. There is a ticket on our community page but its sadly been sitting there on the wishlist for quite a while.
I was responsible for the repository management overhaul of OpenProject 5.0 which introduce a number of improvements to both managing and implementing revision management tools for OpenProject.
If you do have some experience with Ruby, you can use my Gitolite integration plugin for OpenProject as a basis for a new plugin that integrates Mercurial.
The core adapters for Subversion and Git both leverage their command line counterparts to read data on contents, revisions, branches, and so on.
You can also join in at the forums to make that topic more prominent and maybe find a member from the community to work on that integration (e.g., this thread here: https://community.openproject.org/topics/5501?board_id=6&r=5663)