async DB operations

Question

My workflows are hosted in IIS. and each workflow inherits from asynccodeactivity. In BeginExecute, I call command.Beginxxx and in end execute i call EndExecutexxx. I'm using Database Access Block (DAAB).

protected override IAsyncResult BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, object state)
    {
        DbCommand command = null;
        DbConnection dbConnection = null;
        entlib.Database database;

        try
        {
            database = EnterpriseLibraryContainer.Current.GetInstance<entlib.Database>(DatabaseName.Get(context));
            dbConnection = database.CreateConnection();
            command = dbConnection.CreateCommand();
            command.CommandText = CommandText.Get(context);
            command.CommandType = CommandType.Get(context);

            //have removed few assignments here

            context.UserState = new AsyncDbState(database, command);
        }
        catch (Exception e)
        {
            if (command != null)
                command.Dispose();
            if (dbConnection != null)
                dbConnection.Dispose();

            throw e;
        }

        return (database.Beginxxx(command, callback, state));
    }


    protected override TResult EndExecute(AsyncCodeActivityContext context, IAsyncResult iResult)
    {
        TResult result = default(TResult);

        var userState = context.UserState as AsyncDbState;

            try
            {
                result = (TResult)userState.Database.Endxxx(iResult);
            }
            finally
            {
                if (null != userState && null != userState.Command)
                    userState.Command.Dispose();
            }

        return result;
    }

And sporadically it throws error in event log and terminates entire app pool. After Comments by @Will, I did trap inner exception and noticed the actual error happenes

in BeginExecute of a different activity, which inherits from asyncnativeactivity, I have

 var task = AsyncFactory<IDataReader>.Action(() => ExecuteMdxQuery(connectionStringSettings, mdxQuery, commandTimeout, cancellationToken), cancellationToken);                        

                    return AsyncFactory<IDataReader>.ToBegin(task, callback, state);

and AsyncFactory looks like this 

 public static Task<TResult> Action(Func<TResult> actionMethod,CancellationToken token)
    {
        TaskFactory factory = new TaskFactory();
        //TaskFactory factory = new TaskFactory(scheduler);
        return factory.StartNew<TResult>(() => actionMethod(), token);   
        }
  public static IAsyncResult ToBegin(Task<TResult> task, AsyncCallback callback, object state)
    {
        var tcs = new TaskCompletionSource<TResult>(state);
        var continuationTask = task.ContinueWith(t =>
        {
            if (task.IsFaulted)
            {
                tcs.TrySetException(task.Exception.InnerExceptions);
            }
            else if (task.IsCanceled)
            {
                tcs.TrySetCanceled();
            }
            else
            {
                tcs.TrySetResult(task.Result);
            }

An unhandled exception occurred and the process was terminated.

An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1/ROOT/workflowservice

Process ID: 7140

Exception: System.AggregateException

Message: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread.

StackTrace: at System.Threading.Tasks.TaskExceptionHolder.Finalize()

InnerException: Microsoft.AnalysisServices.AdomdClient.AdomdErrorResponseException

Message: Server: The current operation was cancelled because another operation in the transaction failed.

StackTrace: at Microsoft.AnalysisServices.AdomdClient.AdomdConnection.XmlaClientProvider.Microsoft.AnalysisServices.AdomdClient.IExecuteProvider.ExecuteTabular(CommandBehavior behavior, ICommandContentProvider contentProvider, AdomdPropertyCollection commandProperties, IDataParameterCollection parameters) at Microsoft.AnalysisServices.AdomdClient.AdomdCommand.ExecuteReader(CommandBehavior behavior) at WorkflowActivity.AsyncExecuteSafeReader.ExecuteMdxQuery(String connectionStringName, String mdxQuery, Nullable1 commandTimeout, CancellationToken cancellationToken) in d:\B\69\Sources\Infrastructure\WorkflowActivity\AsyncExecuteSafeReader.cs:line 222 at AsyncExecuteSafeReader.ExecuteMdxQuery(String connectionStringName, String mdxQuery, Nullable1 commandTimeout, CancellationToken cancellationToken) in d:\B\69\Sources\Infrastructure\WorkflowActivity\AsyncExecuteSafeReader.cs:line 239 at WorkflowActivity.AsyncExecuteSafeReader.<>c__DisplayClassd.b__a() in d:\B\69\Sources\Infrastructure\WorkflowActivity\AsyncExecuteSafeReader.cs:line 180 at System.Threading.Tasks.Task`1.InvokeFuture(Object futureAsObj) at System.Threading.Tasks.Task.Execute()

Answer 1

The first hint is that this is happening in IIS. While it's usually clear in an application when you're possibly hitting issues caused by multithreading, it isn't so in IIS.

Every request in IIS is serviced by a different thread. Any shared instances are going to be hit by multiple threads. That is often bad news if you're not expecting it.

So my first guess (had to guess because your exception's call stack was cut off; more on that later) was that you're using thread-unsafe code across different threads. I suspected it was centered here EnterpriseLibraryContainer.Current.GetInstance becasue, if that doesn't store the instance per-thread, it'll share the same instance between threads. You'd have to check the code or the docs. Easiest way to test that is to use "make object ID" in the watch window, then compare the results from EnterpriseLibraryContainer.Current.GetInstance within two different threads.

What was clear was that your exception was getting lost because you were re-throwing the exception rather than letting it go. For more information on best practices in this situation, see this answer.

Re-examining the call stack, it still appeared to be a multithreading bug, however it appeared that the problem was that multiple threads were attempting to complete execution of two different Tasks.

Message: The operation completed.
StackTrace: at System.Activities.AsyncOperationContext.ShouldComplete()
(snip)

Something somewhere is attempting to complete execution of the Task but it's already complete. As in, one thread beat another completing the asynchronous operation.

At this point, it was not possible to tell what the actual problem was and where it was happening without the full stack trace of the exception. The best way to get this information is to catch the exception at the first chance within your code and call ToString() on it, or use the "copy to clipboard" link on the exception helper dialog (does the same thing, copies it to the clipboard). This is important because you get the following information

1. Exception type
2. Exception message
3. Stack trace

NOT ONLY for the exception you caught, but for every .InnerException wrapped by this exception!` Often times that's where your real problem is hidden.

And, in this case, when you did that you were able to identify where your code was experiencing reentrancy issues.

Answer 2

@Will, Looks like Mars = false in connection string has resolved it. haven't been able to replicate it. theory is - a query returned result. and end execute was called. but it also returned another resultset? which is why call back was being invoked. but end execute was already called by then. But then again, it's sporadic. If this theory is true, my understanding is that it could have failed all the time. So far haven't been able to crash at all. there were also few procedures which had rowcount on. I'm grateful you took time to comment and share your theory on this. learnt a lot.