I am also having the same issue, I have found the following article somewhat helpful, but still can't seem to get it to work completely. I would have just commented but I don't have enough rep apparently.
http://msdn.microsoft.com/en-us/library/ie/hh772044(v=vs.85).aspx
From what I can tell, this has something to do with IE10 and 11 hijacking the touch action on touch devices in order to allow the user to swipe left and right to switch between apps. I am currently testing on a Surface Pro2 using both the metro version of IE11 and the desktop version of IE11. According to the article you should be able to prevent that behavior using css "touch-action" property but I can't quite get it to work right. I'll update if I find anything more useful.