How to handle a deadlock in third-party code

Question

We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.

We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.

To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.

Like

async Task<bool> WrappedFoo()
{
    var timeout = 10000; 

    var task = Task.Run(() => ThirdPartyCode.Foo());
    var delay = Task.Delay(timeout);

    if (delay == await Task.WhenAny(delay, task ))
    {
        return false;
    }
    else
    {
        return await task ;
    }
}

But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.

Is there a general approach how one should handle deadlocking third-party code?

A CancellationToken won't work because the third-party-api does not provide any cancellation options.

Update: The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings

Have the 3rd Party fix their product ... But I guess you need something "right now"... — Fildor, Mar 30 '21 at 08:41
What is `ThirdPartyCode`? Is it a library? I guess you can try to load it into [separate domain](https://stackoverflow.com/a/6578195/1997232) and if it halts, reload it. — Sinatr, Mar 30 '21 at 08:41
"Is there a general approch how one should handle deadlocking third-party code?" - Yep, stop paying them, stop using their product! — Mitch Wheat, Mar 30 '21 at 08:45
Why use a third party library when .NET already has a Ping class? — Panagiotis Kanavos, Mar 30 '21 at 08:55
@PanagiotisKanavos I updated the question and added some context for the thrid-party method — Ackdari, Mar 30 '21 at 09:03
So, it's SAP. Your company is probably paying a haystack of money on support plans. Use it. — Fildor, Mar 30 '21 at 09:11

score 4 · Answer 1 · answered Mar 30 '21 at 11:08

Is there a general approach how one should handle deadlocking third-party code?

Yes, but it's not easy or simple.

The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").

The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.

This is, unfortunately, the only reliable way to cancel uncancelable code.

it might be useful to note that there are many other ways for the processes to communicate than stdin/stdout. — JonasH, Mar 30 '21 at 15:14
This would be something to an `Erlang` supervisor mechanism. The application isolates itself from the misbehaving code by starting a separate process with the target code and killing it if need arises. — Bercovici Adrian, Apr 01 '21 at 06:52

Bercovici Adrian · Answer 2 · 2021-03-30T11:03:06.407

Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:

public void Caller()
{
     try
     {
          CancellationTokenSource cts=new CancellationTokenSource();
          Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
          Thread.Sleep(3000); //or condition /signal
          cts.Cancel();
     }catch(OperationCancelledException ex)
     {
          //treat somehow
     }
    
}
public void CancellableThirdParty(CancellationToken token)
{
    while(true)
    {
        // token.ThrowIfCancellationRequested()  -- if you  don't treat the cancellation here
        if(token.IsCancellationRequested)
        {
           // code to treat the cancellation signal
           //throw new OperationCancelledException($"[Reason]");
        }
    }
}

As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method , so that the caller just issues the CancellationTokenSource.Cancel.

Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.

A CancellationTokenSource can set a Timeout on the Token(s) it generates or *links*, so you can remove part of the code that *awaits something*. You can also call `[Token].ThrowIfCancellationRequested()`, so other parts of that code can go (in case `Cancel()` is called or can be called before a Timeout). — Jimi, Mar 30 '21 at 10:04
I added the `await Delay` just for an example.I could have had a different thread issue a signal , or basically anything. It was just as a simple example. Also i am aware of the method `ThrowIfCancellationRequested` however i used the flag in order to show that if he needs custom logic on cancellation signal it can be done. — Bercovici Adrian, Mar 30 '21 at 11:03

score -1 · Answer 3 · answered Mar 30 '21 at 08:47

Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :

var cts=new CancellationTokenSource(timeout);

try
{
    await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
    return true;
}
catch(TaskCancelledException)
{
    return false;
}

It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.

That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:

var ping=new Ping();

var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;

The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc

Unfortunately a simple ping won't do the trick (update in question) and a `CancellationToken` does not help since the `Task.Run()` method only checks for cancellation [**before**](https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.task.run?view=net-5.0#System_Threading_Tasks_Task_Run_System_Func_System_Threading_Tasks_Task__System_Threading_CancellationToken_) the work is started but not while executing. — Ackdari, Mar 30 '21 at 09:06

How to handle a deadlock in third-party code

3 Answers3