We experienced similar behaviour recently, with a process being paged out to the swap file, even though there was stacks of RAM still available.
The problem turned out to be the process priority. Firstly in Task Manager it was showing as Below Normal:

However it was more useful to check the Memory Priority from Process Explorer (from SysInternals). This shows on the Performance tab of the process properties:

(A higher number is higher priority here).
When this was below 5, Windows would naturally start to page the process out if it was quiet for about 2 minutes. This was confirmed by watching the Page File usage percentage in Performance Monitor - the green line here:

In our case, the cause was the default priority given to a task by the Task Scheduler in Windows. This defaults to Below Normal, and is also not visible in the UI. Instead you must export the task definition to XML, edit and re-import - as described here.
As that mentions, the default priority given to a task is 7. When that was changed to 4 in the XML, and re-imported:
<Priority>4</Priority>
... this was enough to stop the paging. In Task Manager the priority showed as Normal, in Process Explorer the Memory Priority went from 2 to 5.
(Look out, these numbers go in opposite directions).
It's worth noting, a <Priority>
of 5 or 6 also showed as Normal in Task Manager, but the Memory Priority was still too low to prevent Windows pre-emptively paging.