8

I'm working on a NopCommerce-based project which uses ASP MVC, Autofac and Entity Framework. I'm having exceptions which happen when calling a method on a service from inside an MVC Route which will make a call to the DB using EF.

During dev, everything works fine - however during load testing, when there are concurrent users, 1 or 2 requests will crash out, and one of the following errors are logged to ELMAH.

System.InvalidOperationException ExecuteReader requires an open and available Connection. The connection's current state is open.

System.InvalidOperationException ExecuteReader requires an open and available Connection. The connection's current state is closed.

System.InvalidOperationException The connection was not closed. The connection's current state is connecting.

System.ObjectDisposedException The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.

System.ObjectDisposedException The operation cannot be completed because the DbContext has been disposed.

System.ArgumentException An item with the same key has already been added.

I test this by opening many links on the site and then using a Chrome plug-in to refresh all the tabs, simulating ~25 requests hitting the site at the same time.

The service has 2 methods which are called from inside the route, and then one of these same methods can get called 50+ times from the controller action. Sometimes the exception is triggered from inside the Route, and sometimes it comes from inside the controller. Meaning that the route's GetRouteData has completed, passed over 'flow' to the controller and then failed inside there. But, most of the time, the exception occurs within the Route. When the exception does happen from inside the controller, it's at different lines where the exception occurs.

Sometimes one method will fail, and another time that method will run fine, and then next method in the call stack fails. It's different each time, but these method calls use a common method for retrieving from the DB.

There are 2 other routes which are registered before this one that are mapped to *{url} that perform database lookups of the incoming URL, and the exception never happens there. So, it's not like this route is the first operation to perform any DB work.

The service is dependency registered :-

builder.RegisterControllers(typeFinder.GetAssemblies().ToArray());

builder.Register<IDbContext>(c => new NopObjectContext(DataSettings.DataConnectionString)).InstancePerLifetimeScope();

builder.RegisterGeneric(typeof(EfRepository<>)).As(typeof(IRepository<>)).InstancePerLifetimeScope();

builder.RegisterType<MemoryCacheManager>().As<ICacheManager>().Named<ICacheManager>("nop_cache_static").SingleInstance();

builder.RegisterType<MyService>().As<IMySerivce>()    
               .WithParameter(ResolvedParameter.ForNamed<ICacheManager>("nop_cache_static"))    
               .InstancePerLifetimeScope(); 

The Controller receives the service via Constructor Injection:-

protected readonly IMyService _myService;

public MyController(IMyService myService)
{
    _myService = myService;
}

And the route resolves the service as followed:

public override RouteData GetRouteData(HttpContextBase httpContext)
{
    var _myService = EngineContext.Current.Resolve<IMyService>();
    myService.databaseOperation(); <--- falls over here w/ concurrency
} 

Any ideas why these errors are occurring? and how to resolve them?

From my understanding, it seems that our DBContext is being shared across 2 requests, however in my dependency registration I've told it to resolve as a lifetime score - which should be unique to every request. I've read a lot into causes of this exception and how it's down to the dependency injection framework to control the lifetime of a dependency and to manage disposing of its resources - it's here that things are falling down, it seems.

In regards to the service itself, it works like all other services in the application - there's nothing standout and different about it.

I've pasted the full exception stacktraces http://pastebin.com/XYEwRQsv as well as the offending line of code.

EDIT: I am using MultipleActiveResultSets=True in the connection string. The entity that this service deals with is stand-alone, i.e. it has no relations with other entities, so there should be no multiple iterations of child entities once the query's been executed, as other answers in regards to these exceptions point out.

EDIT: This is the line that throws the exception.

public string GetFriendlyUrlString(string originalUrl)
{
    var friendly =  _cacheManager.Get(originalUrl.ToLower(), () =>
            (from f in _friendlyUrlRepository.ReadOnlyTable      <--------- Here
             where f.OriginalUrl == originalUrl
             select f.Url).ToList().SingleOrDefault());

    return friendly ?? originalUrl;
}

And exception is

ExecuteReader requires an open and available Connection. The connection's current state is closed.

This is so strange. In my route, there are 4 places where a DB call can happen. My exceptions are 95% of the time from one of these 4 calls - usually the first fails, but sometimes the first DB call will be ok and the others fall through. Very infrequently I see the exception come from inside the controller that this route uh...routes to. Again, with that controller exception then the actual DB connection problem happens on one of 5 lines of code - again showing that it's made x many DB calls then fallen over.

Steven Sproat
  • 4,398
  • 4
  • 27
  • 40
  • I can't seem to find the definition of ReadOnlyTable in nopCommerce. It makes me think that the problem that you have some custom code in the repository definition, that could be causing this problem. Could you share that code? – Marco Regueira Nov 13 '14 at 10:59
  • By the way, I've updated your question to add the code you shared at pastebin as it is most relevant here. – Marco Regueira Nov 13 '14 at 11:00
  • Marco - it's something I added to allow the use of 'NoTracking' queries - basically "this.Entities.AsNoTracking();". Cheers – Steven Sproat Nov 17 '14 at 09:16
  • I also researched about joining a no-tracking DbContext onto a tracking one to see whether that could potentially cause it: http://stackoverflow.com/questions/18231039 but the answer points towards it being an okay thing to do. I'm still unsure whether this is a dependency injection problem or down to EF - I'm leaning towards DI due to the contexts being different which makes me think somewhere the per HTTP request dependency injection is falling over? – Steven Sproat Nov 17 '14 at 13:33
  • Have you tried using MVC dependency resolver in the route as in `DependencyResolver.Current.GetService()`? I guess that wont make any difference as you are probably using [`NopDependencyResolver`](https://github.com/nuodb/nopCommerce/blob/master/Presentation/Nop.Web.Framework/Mvc/NopDependencyResolver.cs) which does internally the same thing you are doing. (Unless you have manually configured a different dependency resolver on global.asax?) – Daniel J.G. Nov 17 '14 at 15:11
  • I haven't, no - but I am using the out-of-the-box DI stuff that nop provides. – Steven Sproat Nov 17 '14 at 15:58

4 Answers4

5

Autofac's InstancePerLifetimeScope does NOT guarantee a unique scope per http request. What it guarantees is that there will only be one instance of your component within the lifetime scope from which it was resolved. So, for example, if you resolve an InstancePerLifetimeScope component from the root container, that component will essentially act as a singleton over the course of the application.

If you're resolving your dependency inside a singly-registered service (e.g., a global action filter), then your dependency (dbContext or whatever) will not get disposed on each request -- it'll just stay open, leaking memory and bleeding connections until it either gets disposed or takes your app down.

Assuming you're using the Autofac Mvc integration, as an experiment, perhaps try to TEMPORARILY register your dbContext as InstancePerMatchingLifetimeScope("AutofacWebRequest") -- this is actually the equivalent of doing exactly what Will Appleby said, but without affinity to a newer version of Autofac than you may be using.

If that solves your problem, then basically, @Will Appleby was right. You'll need to register all your InstancePerLifetimeScope components as InstancePerRequest (newly added to Autofac core as of version 3.4.0), or InstancePerHttpRequest (available in Autofac MVC integration prior to 3.4.0).

Otherwise, if I'm guessing correctly, you'll probably get an exception something like this:

DependencyResolutionException: No scope with a Tag matching 'AutofacWebRequest' is visible from the scope in which the instance was requested.

If that happens, you'll still want to change your registrations to InstancePerRequest/InstancePerHttpRequest, but now you have a deeper task at hand. You probably have a singleton (or other object that lives longer than the httpRequest) holding your dbContext hostage. You'll need to identify that longer-lived component, and figure out how to free that captive dependency -- that will likely require refactoring the component to better comply with Autofac's scoping behavior.

You'll want to refer to the troubleshooting section here: http://autofac.readthedocs.org/en/latest/faq/per-request-scope.html#troubleshooting-per-request-dependencies

Here's an excerpt from there:

Common causes for this include:

  • Application registrations are being shared across application types.
  • A unit test is running with real application registrations but isn’t simulating per-request lifetimes.
  • You have a component that lives longer than one request but it takes a dependency that only lives for one request. For example, a singleton component that takes a service registered as per-request.
  • Code is running during application startup (e.g., in an ASP.NET Global.asax) that uses dependency resolution when there isn’t an active request yet.
  • Code is running in a “background thread” (where there’s no request semantics) but is trying to call the ASP.NET MVC DependencyResolver to do service location.
Community
  • 1
  • 1
James Nail
  • 1,541
  • 1
  • 14
  • 23
  • Hey, thanks for the detailed response. RE: "If you're resolving your dependency inside a singly-registered service (e.g., a global action filter) then your dependency will not get disposed on each request " - would this apply to Routes? >90% of the time this is where the exception happens - I have played about with where the Route will resolve the dependency -- receiving it through the constructor (passed over by Resolving it from within global.asax) as opposed to the Route resolving it itself. – Steven Sproat Nov 20 '14 at 12:28
  • I've also checked all services which are registered as SingleInstance and no implementations of these interfaces have any dependencies on any non-singleton implementations (we haven't added anything new here - all default nopCommerce stuff) – Steven Sproat Nov 20 '14 at 12:33
  • Actually, this may very well apply to routes. Can you post your route configuration? Also, which class does the GetRouteData method that you posted belong to? – James Nail Nov 20 '14 at 16:05
  • James: here's how the routing is setup - http://pastebin.com/abTHk30X I've also tried it using this way where I'm passing the Route the service http://pastebin.com/u27hC3vx – Steven Sproat Nov 20 '14 at 16:35
  • Also of note: in the FeaturedRangeRoute and MapCmsPageRoute which come before the "CatalogRoute" (and also mapped to {*url}, they both have similar code for resolving a service, performing a DB call and passing flow to the controller or not, depending if the URL was a match. So, when CatalogRoute is throwing exceptions all over the place, at least 2 DB calls have been made by that point. And sometimes the exception happens in the controller that CatalogRoute sends to...but 90% of the time it's in the route. Very frustrating! – Steven Sproat Nov 20 '14 at 19:38
  • Seeing that the GetRouteData method accepts HttpContextBase, I strongly suspect that a Route instance is long-lived (probably one per application). That means constructor injection is a no-go. As for using service location inside that method, that's a different matter. I'm not at all familar with NopCommerce, or the implementation of the EngineContext that you're using there. The real question there is how that thing integrates with Autofac for lifetime scope management, etc. – James Nail Nov 20 '14 at 19:44
  • You may actually find that you may want to refactor such that you perform your DB calls at a different level in your stack, preferably one where you have clear control over lifetime scope. – James Nail Nov 20 '14 at 19:46
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/65310/discussion-between-steven-sproat-and-james-nail). – Steven Sproat Nov 20 '14 at 19:46
2

Are you sure your IDbContext implementation is being injected with the right scope? I don't use Autofac myself but a quick check of their website documentation suggests that InstancePerRequest would be a more suitable scope for your needs:

InstancePerRequest

Also have you got MARS set to true in your connection string?

Multiple Active Result Sets

Will Appleby
  • 476
  • 4
  • 17
  • 2
    Last versions of nopCommerce (3.30 and 3.40) have removed InstancePerRequest dependencies and replaced them with InstancePerLifetimeScope. – Marco Regueira Nov 18 '14 at 11:35
2

The final fix was down to me having a private variable inside my Route class. I did not realise that Route objects are not instantiated upon every request, and I was initialising this private variable every time inside "GetRouteData". Therefore, during 2 concurrent requests to the site, request A would instantiate the variable and carry on executing its methods, while request B comes in, re-sets the private variable, which messes everything up for request A.

I've now locally scoped this variable and passed it to any other methods that need it and all my problems have gone away. So, it wasn't really Autofac, nor Entity Framework at play - I hadn't considered how MVC treats its routes.

Thanks all for the help and pushing in the right direction of where the problem lay.

Steven Sproat
  • 4,398
  • 4
  • 27
  • 40
1

Your code is essentially correct. The dependencies that you show are from nopCommerce itself (the IDbContext exactly the same but I suppose you have stripped it a bit to show it here) and your service registration is standard. There are more routes with database access that are not failing in nopcommerce; there would be lots of error reports if that was the case..

That made me think that the problem should be somewhere else, so I've conducted the same test as you without finding any problem.

I've never used Glimpse myself, but I've seen it in your stack trace. Have you performed the some test without Glimpse? I see it uses Castle to generate proxies, that could be messing with your dependencies... The problem has to be somewhere else outside nopCommerce's code, maybe in your code or tooling.

BTW. I see you're using nopcommerce 3.30 or 3.40. Since v3.40 MARS is not needed anymore resulting in performance gain...

Marco Regueira
  • 1,045
  • 8
  • 17
  • Great thinking about Glimpse - I removed it and performed my tests (opening 30 tabs within my site and refreshing all) but unfortunately the error still came through about 5 times after several reloads. Unfortunate, I did have a "voilla!" moment as I removed it. I have stripped back nop's data layer to get rid of that DataSettingsManager class or whatever they had, and replaced it with a static property that pulls in the connection string from web.config. – Steven Sproat Nov 18 '14 at 12:17
  • Also disabling MARS triggers this exception pretty frequently: There is already an open DataReader associated with this Command which must be closed first. – Steven Sproat Nov 18 '14 at 12:50
  • You mean that you removed the parameter and it worked? Are you sure that's the reason? Sorry to disagree but it doesn't make sense. It is just a parameter. Are you sure it didn't work after you put down the site clearing completely the application domain? (I've seen that happen before). If you're only changing the connection string source you are not affecting the dependency instantiation. – Marco Regueira Nov 18 '14 at 12:50
  • No sorry, I disabled Glimpse but the error still happened as randomly as before. I was also saying that I stripped back the architecture of Nop's data layer to remove the DataSettingsManager class that previously existed to pull in a connection string from settings.txt, and replaced with a more straightforward class. – Steven Sproat Nov 18 '14 at 12:53
  • Yes but, basically, you're saying that the reason is the connection string or some static class that doesn't play any role in the dependencies... – Marco Regueira Nov 20 '14 at 09:44