In Objective-C the [[UIApplication sharedApplication] canOpenURL:url]
expression may be used to detect whether a URL is handled by any app in the system. If you use PhoneGap or some similar framework, then look for this method. If you have only a webapp, then I'm pretty sure that this is impossible.
The issue with a web app is that it is sandboxed like a web page, it cannot reach outside of the browser. However, it seems to be possible that you can detect the presence with a timing based method. That is, if the app is not installed, the user will return to, or not be able to leave at all the browser within a certain, relatively short time. Thinking along these lines I found this solution: Check if URL scheme is supported in javascript
You may be able to build a solution using this approach, but the "Cannot Open Page" alert box will always be thrown at your users. Though, this IMHO is not really annoying if you handle it correctly on the web app side.