I also hate when the editor suddently goes into visual mode when I accidentally drag with the mouse. I ended up modifying the ideavim source and compiled my own version. I got the idea from another answer https://stackoverflow.com/a/24256022/598781
I just return immediately in the EditorMouseHandler.mouseDragged method. i.e. changed line 1772 in the file ideavim/src/com/maddyhome/idea/vim/group/MotionGroup.java
to:
if (true || !VimPlugin.isEnabled()) return;
This change could have some consequences, but I don't use mouse select anyway.