We have a WCF service hosted on IIS 7.5 which has custom username authentication using a custom membership provider (we authenticate users from a database with some custom logic). The service uses WsHttpBinding with TransportWithMessageCredentials security mode.
It works well with heavy loads but sometimes for about 6-9 minutes it keeps the requests waiting for 6-9 minutes before responding. I added debug logs to the custom authentication module and the service operation that was being called and noticed that for these instances, there was this lag of 6-9 minutes between the end of execution of the authentication module and start of execution of the service method.
The requests responds after these 6-9 minutes correctly. It looks like they got stuck somewhere between the authentication and actual service method call, but I have got no clue as to what is causing it. This is becoming a problem as the service clients are getting timed out during these periods.
Update: I enabled the service trace and observed that for the requests that were getting delayed, there was this delay between the "Received a message over a channel." and the "Understood message header." events. The thread ID also changes between these two events. Can that mean that there are no free threads? But are there no free threads for 7 minutes?
