Scott Hanselman

How to run Background Tasks in ASP.NET

August 26, '14 Comments [42] Posted in ASP.NET | NuGet | NuGetPOW | Open Source
Sponsored By

A few years back Phil Haack wrote a great article on the dangers of recurring background tasks in ASP.NET. In it he points out a few gotchas that are SO common when folks try to do work in the background. Read it, but here's a summary from his post.

  • An unhandled exception in a thread not associated with a request will take down the process.
  • If you run your site in a Web Farm, you could end up with multiple instances of your app that all attempt to run the same task at the same time.
  • The AppDomain your site runs in can go down for a number of reasons and take down your background task with it.

If you think you can just write a background task yourself, it's likely you'll get it wrong. I'm not impugning your skills, I'm just saying it's subtle. Plus, why should you have to?

There's LOT of great ways for you to do things in the background and a lot of libraries and choices available.

Some ASP.NET apps will be hosted in IIS in your data center and others will be hosted in the Azure cloud. The spectrum of usage is roughly this, in my opinion:

  • General: Hangfire (or similar similar open source libraries)
    • used for writing background tasks in your ASP.NET website
  • Cloud: Azure WebJobs 
    • A formal Azure feature used for offloading running of background tasks outside of your Website and scale the workload
  • Advanced: Azure Worker Role in a Cloud Service
    • scale the background processing workload independently of your Website and you need control over the machine

There's lots of great articles and videos on how to use Azure WebJobs, and lots of documentation on how Worker Roles in scalable Azure Cloud Services work, but not a lot about how your hosted ASP.NET application and easily have a background service. Here's a few.

WebBackgrounder

As it says "WebBackgrounder is a proof-of-concept of a web-farm friendly background task manager meant to just work with a vanilla ASP.NET web application." Its code hasn't been touched in years, BUT the WebBackgrounder NuGet package has been downloaded almost a half-million times.

The goal of this project is to handle one task only, manage a recurring task on an interval in the background for a web app.

If your ASP.NET application just needs one background task to runs an a basic scheduled interval, than perhaps you just need the basics of WebBackgrounder.

using System;
using System.Threading;
using System.Threading.Tasks;

namespace WebBackgrounder.DemoWeb
{
public class SampleJob : Job
{
public SampleJob(TimeSpan interval, TimeSpan timeout)
: base("Sample Job", interval, timeout)
{
}

public override Task Execute()
{
return new Task(() => Thread.Sleep(3000));
}
}
}

Built in: QueueBackgroundWorkItem - Added in .NET 4.5.2

Somewhat in response to the need for WebBackgrounder, .NET 4.5.2 added QueueBackgroundWorkItem as a new API. It's not just a "Task.Run," it tries to be more:

QBWI schedules a task which can run in the background, independent of any request. This differs from a normal ThreadPool work item in that ASP.NET automatically keeps track of how many work items registered through this API are currently running, and the ASP.NET runtime will try to delay AppDomain shutdown until these work items have finished executing.

It can try to delay an AppDomain for as long as 90 seconds in order to allow your task to complete. If you can't finish in 90 seconds, then you'll need a different (and more robust, meaning, out of process) technique.

The API is pretty straightforward, taking  Func<CancellationToken, Task>. Here's an example that kicks of a background work item from an MVC action:

public ActionResult SendEmail([Bind(Include = "Name,Email")] User user)
{
if (ModelState.IsValid)
{
HostingEnvironment.QueueBackgroundWorkItem(ct => SendMailAsync(user.Email));
return RedirectToAction("Index", "Home");
}

return View(user);
}

FluentScheduler

FluentScheduler is a more sophisticated and complex scheduler that features a (you guessed it) fluent interface. You have really explicit control over when your tasks run.

using FluentScheduler;

public class MyRegistry : Registry
{
public MyRegistry()
{
// Schedule an ITask to run at an interval
Schedule<MyTask>().ToRunNow().AndEvery(2).Seconds();

// Schedule a simple task to run at a specific time
Schedule(() => Console.WriteLine("Timed Task - Will run every day at 9:15pm: " + DateTime.Now)).ToRunEvery(1).Days().At(21, 15);

// Schedule a more complex action to run immediately and on an monthly interval
Schedule(() =>
{
Console.WriteLine("Complex Action Task Starts: " + DateTime.Now);
Thread.Sleep(1000);
Console.WriteLine("Complex Action Task Ends: " + DateTime.Now);
}).ToRunNow().AndEvery(1).Months().OnTheFirst(DayOfWeek.Monday).At(3, 0);
}
}

FluentScheduler also embraces IoC and can easily plug into your favorite Dependency Injection tool of choice by just implementing their ITaskFactory interface.

Quartz.NET

Quartz.NET is a .NET port of the popular Java job scheduling framework of the (almost) same name. It's very actively developed. Quartz has an IJob interface with just one method, Execute, to implement.

using Quartz;
using Quartz.Impl;
using System;

namespace ScheduledTaskExample.ScheduledTasks
{
public class JobScheduler
{
public static void Start()
{
IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
scheduler.Start();

IJobDetail job = JobBuilder.Create<MyJob>().Build();

ITrigger trigger = TriggerBuilder.Create()
.WithDailyTimeIntervalSchedule
(s =>
s.WithIntervalInHours(24)
.OnEveryDay()
.StartingDailyAt(TimeOfDay.HourAndMinuteOfDay(0, 0))
)
.Build();

scheduler.ScheduleJob(job, trigger);
}
}
}

Then, inside your Application_Start, you call JobScheduler.Start(). There's a great getting started article on Quartz at Mikesdotnetting you should check out.

Hangfire

And last but definitely not least, the most polished (IMHO) of the group, Hangfire by @odinserj. It's a fantastic framework for background jobs in ASP.NET. It's even optionally backed by Redis, SQL Server, SQL Azure, MSMQ, or RabbitMQ for reliability.

The Hangfire documentation is amazing, really. Every open source project's document should be this polished. Heck, ASP.NET's documentation should be this good.

The best feature from Hangfire is its built in /hangfire dashboard that shows you all your scheduled, processing, succeeded and failed jobs. It's really a nice polished addition.

image

You can enqueue "fire and forget" jobs easily and they are backed by persistent queues:

BackgroundJob.Enqueue(() => Console.WriteLine("Fire-and-forget"));

You can delay them...

BackgroundJob.Schedule(() => Console.WriteLine("Delayed"), TimeSpan.FromDays(1));

Or great very sophisticated CRON style recurrent tasks:

RecurringJob.AddOrUpdate(() => Console.Write("Recurring"), Cron.Daily);

Hangfire is just a joy.

Check out the Hangfire Highlighter Tutorial for a sophisticated but easy to follow real-world example.

There's a rich ecosystem out there ready to help you with your background tasks. All these libraries are excellent, are open source, and are available as NuGet Packages.

Did I miss your favorite? Sound off in the comments!


Sponsor: Many thanks to my friends at Raygun for sponsoring the feed this week. I *love* Raygun and use it myself. It's amazing. Get notified of your software’s bugs as they happen! Raygun.io has error tracking solutions for every major programming language and platform - Start a free trial in under a minute!

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

facebook twitter subscribe
About   Newsletter
Sponsored By
Hosting By
Dedicated Windows Server Hosting by ORCS Web
Tuesday, August 26, 2014 9:44:37 AM UTC
I have already done this in other manner
Tuesday, August 26, 2014 10:20:10 AM UTC
Nice post Scott,

Quartz is just amazing, we are also using it for one of our projects.
Milind
Tuesday, August 26, 2014 10:48:02 AM UTC
This is a great write-up, thanks.

I was wondering if these are limited to tasks which are due to run or repeat in the very near future, or can they span over days? Could I schedule a task to run once every evening for an indefinite time? Do all the libraries you've presented support this, or just some of them?
Tuesday, August 26, 2014 11:00:11 AM UTC
Really interesting article.

Is running background tasks in your ASP.NET process the wrong aproach? I would always be very uncomfortable doing this. Off loading tasks to a worker role type process is a much more scalable solution. What happens if you want to load balance your ASP.NET process. Who then becomes responsible for maintaining the background tasks?
Tuesday, August 26, 2014 11:44:11 AM UTC
Is running background tasks in your ASP.NET process the wrong approach? I would always be very uncomfortable doing this.


Yes, there are different problems related to background processing in ASP.NET application itself, some of them described in the given article written by Phil Haack. But almost all of them can be solved by framework itself by using different compensation techniques. Here are the ways how Hangfire solves them (disclaimer: I'm the author).


  • Error handling – logging using your favorite implementation, including Elmah), automatic retries with increasing intervals.

  • Multiple instances (web farms, web gardens) – queues where only one consumer can fetch a job, distributed locks.

  • App domain unloads. This is the most complex thing. In case of graceful termination all jobs are being returned to their queues programmatically. In case of unexpected termination they are being returned after some interval (this interval can be eliminated by using MSMQ or RabbitMQ) – Hangfire guarantees that every queued background job will be processed anyway when there is at least one running Hangfire Server.



Who then becomes responsible for maintaining the background tasks?


As I said earlier, there are automatic retries with increasing intervals, but their number is limited (10 by default, but you can set the number manually). After retry attempts were exceeded, the responsibility for background tasks move to a system administrator (with logging enabled, you will receive notifications about failed jobs) – she can do this task through integrated web interface with ease.
Tuesday, August 26, 2014 12:02:04 PM UTC
Do these solutions handle the issue of the app pool being recycled?

The issue in the passed that i've had, is the app pool gets recycled and the background task code doesn't run until someone visits the websites and starts the process (obviously only really an issue on small infrequently visited websites).

James.
James
Tuesday, August 26, 2014 12:04:05 PM UTC
+1 for quartz.net. I'm using it for scheduled jobs and for long running background tasks submitted from asp.net on demand. http://www.cronmaker.com/ is great for creating those obtuse cron expressions.
ira
Tuesday, August 26, 2014 12:10:13 PM UTC
I am extensively using windows service for most of my healthcare related work. Still looking for elegant distributed architecture, something similar to Celery backed by Queue, AKKA in scala etc., perhaps, Akka.Net may solve this, but need to wait till then windows service with sql server is promising solution for asp.net(I am not considering Azure here).
Tuesday, August 26, 2014 12:12:07 PM UTC
Great article scott!

One thing though, in your WEBBACKGROUNDER examle, you return a cold task (one which hasnt started) via "New Task()". You should probably use "Task.Run" instead to return a hot task
Yuval Itzchakov
Tuesday, August 26, 2014 12:19:00 PM UTC
Nice article!
Even more Glad that the blog referring upworthy isn't appearing at the top any more!! Phew!
Tuesday, August 26, 2014 1:24:57 PM UTC
These are really useful for cache management tasks. Especially if you want to consolidate cache expiry handlers into a single action. This is something the other worker types (azure jobs) and the like cannot handle because they aren't executing in the context of the web application itself.

Mike J
Tuesday, August 26, 2014 1:27:51 PM UTC
Good article.
Using Quartz for building windows service based job scheduler, and its perfectly working. Cron expression are very handy while configuring jobs.
Gourav Saini
Tuesday, August 26, 2014 1:30:14 PM UTC
I lean towards Webjobs, myself. :) Though I have recommended Hangfire to others.

All the libraries here are schedulers for running recurring/periodic background tasks within ASP.NET. There's another category of "background tasks" in ASP.NET that I find people ask for a lot: the ability to "return early" from a request and then have the server do a bit more work.

So, I think HostingEnvironment.QueueBackgroundWorkItem could get an honorable mention. It's a whole level of abstraction beneath these libraries, though; it doesn't do any kind of scheduling or retries, and it's not aware of web farms. It catches (and ignores) errors, and does register the work with ASP.NET, so it's a step above Task.Run without going full-fledged background service.
Tuesday, August 26, 2014 1:39:11 PM UTC
Good article.

I found solution which works best for me for applications installed on own server (not shared hosting). I use c# wrapper for Windows Task Scheduler. Application publishes tasks with default settings into Task Scheduler. UI for configure (for admins) just is there :)

Tasks run specified job to do by executing some REST API within my application: ITaskJob { Execute(); }.

Best regards
dariol
Tuesday, August 26, 2014 1:43:46 PM UTC
What are peoples' thoughts on HostingEnvironment.QueueBackgroundWorkItem? I've been using this successfully for quite some time now.
Ryan Besko
Tuesday, August 26, 2014 1:49:23 PM UTC
The approach I've used to this problem is to do my background processing in a Windows service, and then use polling or a message queue to find out about tasks. To my surprise, that didn't even make the list.

Is there a reason why a service is undesirable?
Blair Learn
Tuesday, August 26, 2014 2:15:37 PM UTC
.NET 4.5.2 introduces HostingEnvironment.QueueBackgroundWorkItem to help run background tasks in an asp.net app domain. The method registers the task with asp.net so that the runtime will know about it during recycles or app shutdowns, and it gives you a CancellationToken that will cancel whenever an event like this is triggered.

The method mitigates many of the risks of trying to run a background task in-process in an asp.net app domain, but if you want fancy scheduling, repetition or durability you're still better off using a helper library like Hangfire.
Tuesday, August 26, 2014 3:47:02 PM UTC
"(or similar similar open source libraries)"

Those two words are similar enough that the second one (or the first one?) is redundant.
Tuesday, August 26, 2014 4:20:36 PM UTC
I've actually been doing this in a fairly reliable manner for many years, and it's not very sophisticated. You call StartServices() from global.asax in the app start, and off you go:

https://github.com/POPWorldMedia/POPForums/blob/master/PopForums/Web/PopForumsActivation.cs

Someone once asked me, "Well what happens if the app shuts down due to inactivity?" To which I responded, "Then there isn't any work for the background stuff to do." :)

I've already done some work to run this in an Azure worker role (code is already in the repo, sans any instrumentation), but the original intent of this was to serve people with cheap hosting who couldn't run anything outside of the context of the site, because they had no access.
Tuesday, August 26, 2014 4:43:52 PM UTC
If you need something simple, why not just run a task ? What value is added by say, WebBackgrounder ?
Tuesday, August 26, 2014 5:10:58 PM UTC
Thank you very much for this article. I was struggling with this for quite a while. My plan was to create a Windows Service that runs a specific action on the website to avoid complications. How are these solutions better/different than creating a Windows Service? Any thoughts?
Tuesday, August 26, 2014 5:25:00 PM UTC
I had the same issues @james speaks of. In the end I ended up using Pingdom to hit an MVC action every so often.

It's not ideal, but proved more reliable than the alternative. May have to check out hangfire though!
Tuesday, August 26, 2014 6:11:44 PM UTC
Nice summary of the options. I'm leaning towards WebJobs these days, but Hangfire does look interesting. One issue I had with Quartz scheduling in ASP.NET was that I did not take into consideration a web farm scenario. I published to Azure Websites (before WebJobs was out), and it took me a bit to realize my background scheduled tasks were running three places. Even though I had a single instance of the Website running, apparently MS keeps one or two running off to the side. For queue processing, this was desirable. But in some cases- database cleanup for example- you want to guarantee only one instance of a job is going to run.

Just an example of one of those nuances a naive approach like mine may overlook...
Tuesday, August 26, 2014 6:46:22 PM UTC
@Daniel: Aren't Webjobs also running per-instance? That's fine if it's just against a queue, but potentially runs into issues with other sources (SQL and such).
Tuesday, August 26, 2014 7:10:03 PM UTC
Not applicable to all projects, but worth mentioning: Orchard has a built-in job scheduler.
Tuesday, August 26, 2014 7:30:54 PM UTC
too bad Hangfire is .Net 4.5 only :-(. We are months away form being able to upgrade
Tuesday, August 26, 2014 8:39:04 PM UTC
+1 to Hangfire. Been using it for my latest project and it works like a charm!
Tuesday, August 26, 2014 8:45:53 PM UTC
Hangfire depend on Microsoft.Owin.Host.SystemWeb 2.1, then I have other library depend on Microsoft.Owin.Host.SystemWeb 1.0 and cannot switch out for specific reason. Start to hate Owin implementation lately because of dependency on Katana project.

Whenever I see a new library come out with Owin implementation, it can be pretty much ignored or try my best to avoid using it.
Jason
Tuesday, August 26, 2014 10:48:28 PM UTC
James and FJ - Yes, none of the libraries actually give a *guarantee* that the tasks will complete. Also, since ASP.NET 4.5.1 has the ability to suspend all threads in an AppDomain, there is no guarantee that background tasks will complete unless a request comes in to wake up the suspended thread. This is per Levi on our team.
Scott Hanselman
Wednesday, August 27, 2014 9:25:38 AM UTC
Hangfire has really good documentation, build with Sphinx and hosted on ReadTheDocs.org (I sort of know it's going to be good when I see this combination).
Tomas
Wednesday, August 27, 2014 12:35:33 PM UTC
If the Hangfire Highlighter Tutorial was calling HighlightSource method in IIS-hosted WCF service in a new thread (say using just new Thread(HighlightSource)) should it still use Hangfire instead of new Thread to make sure that job will finish before app domain is shut down?
What about WCF
Wednesday, August 27, 2014 8:47:37 PM UTC
Hangfire looks awesome. I've generally set up a queue in SQL Server and have a console app pick up and perform whatever task I wanted, but that's kind of a pain and not the most elegant.
Thursday, August 28, 2014 5:39:24 AM UTC
Why would you even want to run a task in the process of the website?
This seems like a last resort if you don't have access or aren't able to run console/service-like process that can process a queue. The risk seems to great to me to use this just so it's all in the same project.
Thursday, August 28, 2014 6:50:48 AM UTC
Another vote for Hangfire.

As someone who's spent years building apps using good old system.threading.tasks, Hangfire is one of those annoying (in a good way) frameworks that make you scream 'Why didn't I see this months ago?' :-)
Thursday, August 28, 2014 1:32:07 PM UTC
I'm usually not a fan of shoe-horning things into where they shouldn't be, but I can see a genuine need for this.

I intended to just skim the documentation, but ended up reading it all as it was so well written.

Great idea, I will be using it myself and will report back.
Will
Thursday, August 28, 2014 9:25:34 PM UTC
Thanks for another great article Scott. I sure will check Hangfire next time.

p.s. Can you 'fix' the second paragraph after the WEBBACKGROUNDER h3?
John
Friday, August 29, 2014 7:59:34 AM UTC
Work or Task Queues are a common pattern for handing off long running work to a separate process. Most of the 'BUS' implementations out there NServiceBUS, FubuBUS, ReBUS, and Mass Transit can be used in this way. Outside the .NET space Celery is a Python library that has similar characteristics. Some support 'scheduled' or 'delayed' jobs, others don't.

My own OSS project Brighter is not only a Command Processor/Dispatcher, it also supports Work Queues over .NET, using RabbitMQ today, although I am looking at an experimentat RESTMS application layer support as well. Always happy for feedback: http://iancooper.github.io/Paramore/Introduction.html
Monday, September 01, 2014 6:32:36 AM UTC
Hi Scott,

Thanks for the interesting post.

Question - In Hangfire or any of the libraries, can I queue a call to a service method that uses TPL. I am trying to re-use existing service methods that use async await but I also want to perform that work in a background thread in some cases.

Example -
/* question - guess this wont be using the async await feature and will just queue the service call as a blocking sync call on the background thread and that thread wont be avail other work whilst its waiting on I/O in the service method? */

BackgroundJob.Enqueue<Task>(() => BusinessService.DoWorkAsync(user));


thanks
Jatin
Jatin
Tuesday, September 02, 2014 2:53:49 PM UTC
What about simple windows service? It is not the best one, but I think many people use it for the background tasks implementation.
Alexander Skogorev
Friday, October 10, 2014 7:38:10 PM UTC
Hi Scott

You said:
--------------------------------
James and FJ - Yes, none of the libraries actually give a *guarantee* that the tasks will complete. Also, since ASP.NET 4.5.1 has the ability to suspend all threads in an AppDomain, there is no guarantee that background tasks will complete unless a request comes in to wake up the suspended thread. This is per Levi on our team.
--------------------------------
First. What does "This is per Levi on our team?". 'm Not American and do not know this expression

And is there any date, any experança so we can have some feature in asp.net to ensure that the tasks actually execute.

I need it, but without having to change any settings in IIS.
An optical solution would only put in one web.config "alwaysRuning = true" setting and ready

Please tell me that we will have news this direction
penihel
Sunday, October 19, 2014 3:43:38 PM UTC
What's up, all is going nicely here and ofcourse every one is
sharing facts, that's truly fine, keep up writing.


My blog ... games for kids
Saturday, October 25, 2014 12:56:13 PM UTC
You da MAN, Hanselman!! x
Nadz
Name
E-mail (will show your gravatar icon)
Home page
 
Comment (Some html is allowed: a@href@title, b, blockquote@cite, em, i, li, ol, pre, strike, strong, sub, super, u, ul) where the @ means "attribute." For example, you can use <a href="" title=""> or <blockquote cite="Scott">.
Live Comment Preview

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.