Sorry, but this is a large problem with having a huge singleton (unity) around implementing IDependencyResolver. I would bet that you are leaking memory.
edit
In response to your comment:
The reason that memory leak or the DI Container struck me as the issue is because there should be really no time inbetween the controller firing up and an action firing up as they are very close to each other. A simple way to test if it is a memory leak is to let the application sit unentered for a good amount of time (30 minutes to 2 hours) and then attempt to revisit it. If this is quick at first then that could indicate a memory leak. If it is slow on first request then perhaps it is something else. If memory leaking is not the issue then perhaps it is something easier. You said it is before the controller finishes so I would rule out rendering the view (which can take some time). Something you said makes me wonder about your web.config
file. "this happens in all environments from development to production." Perhaps your production environment is still running under debug=true
. These are all the ideas I could think of at the moment.