2

I have an MVC and WebAPI application that needs to log activities performed by the users back to my database. This is almost always a single insert into a table that have less than 5 columns (i.e. very little data is crossing the wire). The data interface that I am currently using is Entity Framework 6

Every once in a while, I'll get a large number of users needing to log that they performed a single activity. In this case, "Large Number" could be a couple hundred requests every second. This typically will only last for a few minutes at most. The rest of the time, I see very manageable traffic to the site.

When the traffic spikes, Some of my clients are getting timeout errors because the page doesn't finish loading until the server has inserted the data into the database. Now, the actual inserting of the data into the database isn't necessary for the user to continue on using the application, so I can cache these requests somewhere locally, and then batch insert them later.

Is there any good solutions for ASP.NET MVC to buffer incoming request data and then batch insert them into the database every few seconds?

As for my environment, I have several servers running Server 2012 R2 in a load balanced Web Farm. I would prefer to stay stateless if at all possible, because users might hit different servers per request.

joe_coolish
  • 7,201
  • 13
  • 64
  • 111
  • Somehow, I'm thinking _"this is like analytics tracking (events, etc)"_ so my brain goes that direction - is anything preventing a client side approach? Collecting data and persisting on client until "some time" you define to "batch out"? Kind of a "poor (wo)man/s message queue" :) Or, if you think about it, its like a "client side shopping cart" (add, remove, flush as needed). – EdSF Sep 11 '15 at 23:45

1 Answers1

2

When the traffic spikes, Some of my clients are getting timeout errors because the page doesn't finish loading until the server has inserted the data into the database.

I would suggest using a message queue. Have the website rendering code simply post an object to the queue representing the action, and have a separate process (e.g. Windows Service) read off the queue and write to the database using Entity Framework.

UPDATE

Alternatively you could log access to a file (fast), and have a separate process read the file and write the information into your database.

I prefer the message queue option, but it does add another piece of architecture.

Eric J.
  • 147,927
  • 63
  • 340
  • 553
  • What kind of message queue? MSMQ is a technology that I'm not really familiar with, so if there is another queue service that runs on top of IIS (or Server 2012 R2 for that matter) I would be very interested in learning about it. btw, I have multiple servers in the web farm, so if this can be stateless, that would be ideal for my setup. I'll add that to my question – joe_coolish Sep 11 '15 at 20:43
  • Note: If you are OK with very occasionally losing some of the user activity logging, you can use a separate thread and something like a BlockingCollection() rather than a message queue. Unprocessed data will be lost when the app domain recycles http://stackoverflow.com/questions/10852515/correctly-implement-background-process-thread-in-asp-net – Eric J. Sep 11 '15 at 22:25
  • I actually ended up moving the high transaction data from SQL over to Redis. There is little or no relationship in that data, so I was able to rewrite everything pretty quickly. It took a lot more thinking to get the data in, but man is it fast! – joe_coolish Nov 08 '15 at 20:33