20

I have a webapi that is designed to process reports in a queue fashion. The steps the application takes are as follows:

  • Receive content
  • Map the content to an object and place it into the queue
  • Poll for pending items in the queue
  • Process items in queue one at a time

I was thinking to use Entity Framework to create a database of queued items, such as:

public class EFBatchItem
{
    [Key]
    public string BatchId { get; set; }
    public DateTime DateCreated { get; set; }
    public DateTime DateCompleted { get; set; }
    public string BatchItem { get; set; }
    public BatchStatus Status { get; set; }
}

My question - Is there a more efficient way, using NServiceBus, BlockingCollection or ConcurrentQeueue, than to constantly poll the database and pull out pending items one by one? I have not used queues before.

One thought is to create a queue of tasks, and on a separate thread process all pending tasks. Somewhat similar to Most efficient way to process a queue with threads but I want to ensure that I am going the most efficient route.

EDIT: A big question I have here is the best way to display the progress to the user. Once the user submits content, he gets taken to a new page and can view the status by the batch identifier. Is MSMQ necessary, or NServiceBus, in order to notify the user? This seems like a variation of the REquest/Acknowledge/Push paradigm?

Community
  • 1
  • 1
appsecguy
  • 1,019
  • 3
  • 19
  • 36
  • my solution would be a `lock` statement around the part of code where you write to the database. – efkah Feb 05 '13 at 15:26
  • Thanks for the tip - hadn't thought of that, and would probably have caused a lot of issues later on :) Any thoughts on the method of queuing? Should I bother trying a BlockingCollection, or just the constant database polling? Updated the question with a quick edit – appsecguy Feb 05 '13 at 15:33

4 Answers4

28

IMHO, your ASP.NET Web API application shouldn't run those background tasks by itself. It should only be responsible to receive the request, stick it inside the queue (as you indicated) and return the response indicating the success or failure of the received message. You can use variety of messaging systems such as RabbitMQ for this approach.

As for the notification, you have a few options. You can have an endpoint which the client can check whether the processing is completed or not. Alternatively, you could provide a streaming API endpoint which your client can subscribe to. This way, the client doesn't have to poll the server; the server can notify the clients which are connected. ASP.NET Web API has a great way of doing this. The following blog post explains how:

You can also consider SignalR for this type of server to client notifications.

The reason why background tasks are hard for ASP.NET Web API application is that you're responsible to keep the AppDomain alive. This is a hassle especially when you are hosting under IIS. The following blog posts explains really good what I mean:

David Gardiner
  • 16,892
  • 20
  • 80
  • 117
tugberk
  • 57,477
  • 67
  • 243
  • 335
  • Well, the background task is essentially running several ASP.NET functions to generate a report automatically. Would you then recommend a separate application, and to pass the content to that? – appsecguy Feb 05 '13 at 16:26
  • 1
    @pendraggon87 what do u mean by "ASP.NET functions"? – tugberk Feb 05 '13 at 16:27
  • Essentially, I am receiving content, and then using the Interop libraries to generate a word document. I had made a web application that allowed users to generate the document via a button click after inputting a lot of information, but I am working now on just making a system where I can generate the report by receiving JSON data. So I have all my code written to write the report. Are you recommending I put that code in a separate application? – appsecguy Feb 05 '13 at 16:35
  • @pendraggon87 what I would do is that I could get the data through the my ASP.NET Web API application, stick the details into the queue and then let the workers generate the word doc. for notifications, I would pick one of the above described options. – tugberk Feb 05 '13 at 16:40
  • Thanks! And would you recommend using BackgroundWorker for the workers? (I have not had much experience at all in dealing with queues, etc. I currently just have my EF database of items and using the database as a queue based on which rows have the Pending status. – appsecguy Feb 05 '13 at 16:41
  • @pendraggon87 well, do your research and decide based on that, not on my word :) but RabbitMQ is a strong candidate. It has an official .NET client and one community driven client (EasyNetQ). – tugberk Feb 05 '13 at 16:44
  • I'd love some feedback on a service I've been working on to run/queue background tasks. http://sharphooks.net It involves implementing a single method of an interface, uploading your files via a web UI, then posting new jobs via an HTTP POST. – cdeutsch Jul 02 '13 at 18:14
7

It's a NuGet package and it's called HangFire - https://github.com/HangfireIO/Hangfire. The tasks persist even beyond apppool recycling.

Kind Contributor
  • 17,547
  • 6
  • 53
  • 70
  • thanks for this link, I've never come across it before but could be just what I need! – Phil Mar 23 '16 at 13:18
2

I like @Todd's solution. Just for the sake of completeness, but there's another option not mentioned yet:

HostingEnvironment.QueueBackgroundWorkItem(cancellationToken =>
{
    // Some long-running job
});

Note:

"When ASP.NET has to recycle, it will notify the background work (by setting a CancellationToken) and will then wait up to 30 seconds for the work to complete. If the background work doesn’t complete in that time frame, the work will mysteriously disappear."

And avoid using service methods with an injected DbContext here as this won't work.

Sources: MariusSchulz and StephenCleary

Anytoe
  • 1,605
  • 1
  • 20
  • 26
1

If you use Asp.net Core,you can use IHostService.

menxin
  • 2,044
  • 1
  • 17
  • 15