Rebus crash in .NET async core

Question

I am trying to upgrade an existing website from Rebus 0.45 to Rebus 2.0 and have run into a problem where System.Web.ThreadContext.AssociateWithCurrentThread crashes with a null reference. I have absolutely no idea about why, so I am looking for possible causes that I can look for.

The only trace I have is the Windows Event Log that registers the following when trying to send a message:

An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1/ROOT/website

Process ID: 14460

Exception: System.NullReferenceException

Message: Object reference not set to an instance of an object.

StackTrace:    at System.Web.ThreadContext.AssociateWithCurrentThread(Boolean setImpersonationContext)
   at System.Web.HttpApplication.OnThreadEnterPrivate(Boolean setImpersonationContext)
   at System.Web.LegacyAspNetSynchronizationContext.CallCallbackPossiblyUnderLock(SendOrPostCallback callback, Object state)
   at System.Web.LegacyAspNetSynchronizationContext.CallCallback(SendOrPostCallback callback, Object state)
   at System.Web.LegacyAspNetSynchronizationContext.Post(SendOrPostCallback callback, Object state)
   at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.PostAction(Object state)
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task& currentTask)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.AwaitTaskContinuation.<>c.<ThrowAsyncIfNecessary>b__18_0(Object s)
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

The website is running ASP.NET MVC 3.0 with a Castle Windsor dependency injector and uses Rebus.SqlServer for transport setup as oneway client:

var busConfigurer = Configure.With(new CastleWindsorContainerAdapter(container))
  .Logging(l => l.Use(new RebusLoggerFactory()))
  .Routing(t => t.TypeBased().AddEndpointMappingsFromAppConfig())
  .Subscriptions(s => s.StoreInSqlServer(dbConnectionString, "RebusSubscriptions_v2"))
  .Transport(t => t.UseSqlServerAsOneWayClient(dbConnectionString, "RebusMessages_v2"))
  Start();

IIS is setup for ".NET 4.5" application pool.

The web site starts and the bus logs:

- Supplied connection string will be modified to enable MARS
- Database already contains a table named 'RebusMessages_v2' - will not create anything
- Supplied connection string will be modified to enable MARS
- Starting periodic task 'CleanupTrackedErrors' with interval 00:01:00
- Starting bus 1
- Started

At a later time a message is sent via Rebus that logs:

- Sending SubmissionCreatedEvent -> SelfService.Service.input

... and boom! the whole application crashes.

Before sending the message the application has interacted with NHibernate to store details about the operation.

Update

By changing Bus.Send(msg) to Bus.Send(msg).Start() I can trigger the same error synchronously and get the following stack trace when debugging in Visual Studio:

Exception thrown: 'System.NullReferenceException' in System.Web.dll
Exception thrown: 'System.NullReferenceException' in mscorlib.dll
System.Transactions Critical: 0 : 
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Critical"><TraceIdentifier>http://msdn.microsoft.com/TraceCodes/System/ActivityTracing/2004/07/Reliability/Exception/Unhandled</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>/LM/W3SVC/1/ROOT/website-2-131225642632547395</AppDomain>
<Exception><ExceptionType>System.NullReferenceException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Object reference not set to an instance of an object.</Message><StackTrace>   at System.Web.ThreadContext.AssociateWithCurrentThread(Boolean setImpersonationContext)
   at System.Web.HttpApplication.OnThreadEnterPrivate(Boolean setImpersonationContext)
   at System.Web.LegacyAspNetSynchronizationContext.CallCallbackPossiblyUnderLock(SendOrPostCallback callback, Object state)
   at System.Web.LegacyAspNetSynchronizationContext.CallCallback(SendOrPostCallback callback, Object state)
   at System.Web.LegacyAspNetSynchronizationContext.Post(SendOrPostCallback callback, Object state)
   at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.PostAction(Object state)
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task&amp;amp; currentTask)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.AwaitTaskContinuation.&amp;lt;&amp;gt;c.&amp;lt;ThrowAsyncIfNecessary&amp;gt;b__18_0(Object s)
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()</StackTrace><ExceptionString>System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Web.ThreadContext.AssociateWithCurrentThread(Boolean setImpersonationContext)
   at System.Web.HttpApplication.OnThreadEnterPrivate(Boolean setImpersonationContext)
   at System.Web.LegacyAspNetSynchronizationContext.CallCallbackPossiblyUnderLock(SendOrPostCallback callback, Object state)
   at System.Web.LegacyAspNetSynchronizationContext.CallCallback(SendOrPostCallback callback, Object state)
   at System.Web.LegacyAspNetSynchronizationContext.Post(SendOrPostCallback callback, Object state)
   at System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.PostAction(Object state)
   at System.Threading.Tasks.AwaitTaskContinuation.RunCallback(ContextCallback callback, Object state, Task&amp;amp; currentTask)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.AwaitTaskContinuation.&amp;lt;&amp;gt;c.&amp;lt;ThrowAsyncIfNecessary&amp;gt;b__18_0(Object s)
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()</ExceptionString></Exception></TraceRecord>

Update II

Replacing Bus.Publish(msg) with bus.Advanced.Routing.Send("SelfService.Service.input", msg) works. So apparently there is something wrong in the subscription/routing stuff as Send(...) bypasses the that.

Update III

Apparently adding <httpRuntime targetFramework="4.6" ... in the web.config solved the problem (so far, it still seems a bit erratic). The web.config already had <compilation ... targetFramework="4.6">:

  <system.web>
    <httpRuntime targetFramework="4.6" maxRequestLength="5000" requestValidationMode="2.0" />
    <compilation debug="true" targetFramework="4.6">
      <assemblies>...</assemblies>
    </compilation>
    <pages validateRequest="false">
      <namespaces>...</namespaces>
    </pages>
  </system.web>

When I googled for "nullreferenceexception onthreadenterprivate" I got to [this other SO post](http://stackoverflow.com/questions/12969071/async-logging-throwing-a-nullreferenceexception) that suggested that the exception occurs when you forget to `await` something async..... could it be so simple that you forgot to `await Bus.Send(yourMessage)`? — mookid8000, Nov 02 '16 at 13:24
There is no "await" since it gives me "The 'await' operator can only be used within an async method." and none of the existing code is async. All this async stuff is new to me so any help is appreciated :-) — Jørn Wildt, Nov 02 '16 at 13:56
Seems like ASP.NET has some issues with calling async from sync. See for instance http://stackoverflow.com/questions/12701879/confusing-behaviour-when-invoking-async-methods-inside-asp-net — Jørn Wildt, Nov 02 '16 at 14:25
Regarding Update II: When you send explicitly to one specific queue, it bypasses a router lookup (which is async), which means that it can behave differently if async/await is causing issues. — mookid8000, Nov 02 '16 at 20:43
But you should (almost) always either `await` functions that return `Task`s, or `.Wait()`/`.Result` them to wait for their completion (and possibly catch their exceptions)... only thing is: ASP.NET wants your continuations to run on the calling thread, so `.Wait()` will deadlock — mookid8000, Nov 02 '16 at 20:45

score 4 · Accepted Answer · edited Jan 09 '17 at 23:52

4

The fix turned out to be to make sure the website is running .NET 4.5 or higher:

<system.web>
  <httpRuntime targetFramework="4.6" />
  <compilation targetFramework="4.6">
    ...
  </compilation>
</system.web>

Apparently the ASP.NET team made ASP.NET more task aware in .NET 4.5.

edited Jan 09 '17 at 23:52

brnis

13
8

answered Nov 03 '16 at 13:05

Jørn Wildt

4,274
1
21
31

Rebus crash in .NET async core

1 Answers1