keep load balancer from timing out during long operation

Question

I'm using ASP.NET MVC 4 to accept a video upload of up to 30 mb in size. Once the video is uploaded to the site, the video is then posted to a 3rd party web service using a HttpWebRequest. This upload to the 3rd party service must complete and return a response before my site can return a response to the browser. On Rackspace Cloud Sites, the load balancer has a 30 second timeout. If no response is returned from my site through the load balancer for 30 seconds, then load balancer returns a timeout error to the browser and the request is terminated.

So what I've been told needs to happen is that my site needs to keep the connection alive by sending data back to the browser while my site is sending the video to the 3rd party system, so that the load balancer doesn't timeout. How would you accomplish this task?

I'm looking for 2 things - What do I send back as data to the browser while the upload is happening? My actual response is JSON, so if I can keep my regular JSON response at the end, that would be ideal.

And finally, how do I execute my HttpWebRequest upload to the 3rd party while sending the keep-alive data at the same time?

-

For reference, this is the load balancer connection timeout doc from rackspace cloud: http://www.rackspace.com/knowledge_center/article/connection-timed-out-error-message-1

They don't offer up any concrete solutions for this problem.

If you need a quick and dirty solution that keeps the rackspace load balancer from timing out, you can use my answer below. This is a complete hack and not robust enough to handle much of a load at all. The ideal solution would be to kick off the long executing operation, return that request to the browser, and then have some sort of polling from the browser that checks to see if the operation has finished. That's not how my solution works. If you happen to write your own solution that uses polling, please post it on this thread. Thanks! — Rafe, Oct 14 '12 at 16:26
Rackspace just announced that they are adding a feature so that you can set the timeout on the load balancer. See here: http://feedback.rackspace.com/forums/71021-product-feedback/suggestions/2203103-create-a-timeout-config-to-loadbalancers?tracking_code=4ae2f17a9c6d1f4d0077d48085775091 — Rafe, Nov 09 '12 at 13:23

Rafe · Accepted Answer · 2012-10-10T15:50:50.073

Here's how I solved my problem. This solution should be applicable to any .net hosting that uses a load balancer with an unreasonable timeout. Rackspace Cloud Sites have a 30 second timeout on their load balancer. Amazon and Azure have longer timeouts, but this still might be a useful solution if you happen to have a long-running operation.

In asp.net mvc 4 you can change your controller to inherit from AsyncController. This allows you to spin off async tasks and then await their completion. This solution creates a "keep alive" task which sends data back through the asp.net Response every 10 seconds so that the load balancer detects activity and doesn't timeout. This task runs until it sees that the uploadFinished variable is set to true.

The other task performs the long-running upload, or whatever task needs to be completed before the asp.net mvc controller ends the web request it is handling. After the long-running operation is completed it sets the uploadFinished variable to true.

Finally, I'm manually constructing a container json object around the actual json response so that I can have the "keep alive" data be part of valid json that gets sent back to the browser. A slightly ugly hack, but it works!

using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Web.Mvc;
using System.Web.Script.Serialization;

namespace MvcApplication4.Controllers
{
    public class ExampleController : AsyncController
    {

        [HttpPost]
        public string Upload(UploadModel model, HttpPostedFileBase videoFile)
        {
            // when using jquery ajax form for upload
            // IE requires text/plain content type
            // otherwise it will try to open the response as a file
            // instead of pass the response back to your javascript
            if (Request.AcceptTypes.Contains("application/json"))
            {
                Response.ContentType = "application/json";
            }
            else
            {
                Response.ContentType = "text/plain";
            }

            // start json object
            Response.Write("{\"keepalive\":\"");
            Response.Flush();

            Task[] tasks = new Task[2];
            tasks[0] = Task.Factory.StartNew(() => DoKeepAlive());
            tasks[1] = Task.Factory.StartNew(() => DoUpload(model, videoFile));
            Task.WaitAll(tasks);

            // end keepalive json property
            Response.Write("\",\"data\":");

            // insert actual response data
            JavaScriptSerializer json = new JavaScriptSerializer();
            Response.Write(json.Serialize(uploadResponse));

            // end json object
            Response.Write("}");

            return "";
        }

        public bool uploadFinished = false;
        public UploadResponseModel uploadResponse = null;

        public void DoUpload(UploadModel model, HttpPostedFileBase videoFile)
        {
            // do upload to 3rd party
            MyServiceClient c = new MyServiceClient();
            uploadResponse = c.UploadVideo(model, videoFile);
            uploadFinished = true;
        }

        public void DoKeepAlive()
        {
            // send . every 10 seconds
            while (!uploadFinished)
            {
                Response.Write(".");
                Response.Flush();
                Thread.Sleep(10000);
            }
        }
    }
}

This solution depends on .Net 4 to do the async stuff. If you have .Net 4.5 there are newer ways to do async in your mvc controllers that don't depend on the AsyncController class.

This site is helpful and explains how to do async operations in asp.net mvc in various versions of the .Net framework:

http://dotnet.dzone.com/news/net-zone-evolution

I wouldn't advice you to implement this solution. You are jeopardizing precious worker threads on your server during the upload operation which could quickly cripple it down especially if you have many clients. A much better solution is to start the upload of the file and return immediately. Then the client could poll from time to time for the completion of the task. And if you use SignalR it could even be the server that pushes back to the client to notify it that the task has finished. — Darin Dimitrov, Oct 10 '12 at 15:15
Thankfully this is an action on my site that doesn't get called often. I think if you use the newer async options in .Net 4.5 the worker threads can be put back into the thread pool while the async operations are executing on non-thread-pool threads. Unfortunately Rackspace Cloud Sites does not support .Net 4.5 at this time. — Rafe, Oct 10 '12 at 15:46
No, no threads will be ever put back. If you put back worker threads the Response object dies along with them (which is the whole point of IO Completion Ports) and when the Response object dies you will of course get NREs when you attempt to write to it. There's no solution to this problem other than having the client poll for completion. Of course you could always go with your solution, I am just mentioning it here for people having the same problem about the risks with this solution. It's good that in your case this action is not called very often, but personally I would never do that. — Darin Dimitrov, Oct 10 '12 at 15:49
Check out the response by Stephen Cleary on this post: "The really cool thing about async is [...] While the download is going, the await actually returns out of the async method, leaving the request thread. That request thread is returned back to the thread pool - leaving 0 (zero) threads servicing this request." http://stackoverflow.com/a/9229660/27497 — Rafe, Oct 10 '12 at 15:54
Of course that's true only if you are using IO Completion Ports. In the example he provided he uses a DownloadDataTaskAsync on the WebClient. In your case you are spinning a new task in which you are sleeping and even worst you are waiting for it to complete in the main thread. So while his solution would greatly apply to downloading the file from the remote location, if you start pinging the client at regular intervals you are loosing the whole point of it. — Darin Dimitrov, Oct 10 '12 at 15:56

score 1 · Answer 2 · edited May 23 '17 at 12:26

1

Too bad you can't adjust the load-balancer config for your site.

I think you want to do a javascript keep-alive.

For example:

(function poll(){
$.ajax({ url: "server", success: function(data){
    //Update your dashboard gauge
    salesGauge.setValue(data.value);

}, dataType: "json", complete: poll, timeout: 30000 });
})();

References:

keep load balancer from timing out during long operation

2 Answers2

Linked