2

I haven't seen this answered yet, pretty simple setup...

Question: I'm getting a deadlock, I want to know how to avoid it, given my setup.

I have ServiceA, which implements IServiceA, and ServiceB, which implements IServiceB. Both services are marked as "perSession" and "re-entrant"...

IServiceA supports this method:

DataA GetDataA( );

IServiceB supports this method:

DataB GetDataB( );

in my app, I call ServiceA's GetDataA( ) method, and inside ServiceA's GetDataA method, it opens up a proxy connection to ServiceB, and calls GetDataB( ).

GetDataB on the service end (not the client's end) executes correctly (debug log says so) but when it tries to return the data to ServiceA, it deadlocks.

I'm assuming this is because when I called into ServiceA from "user", it obtained the IO completion port "delivery lock" (or perhaps it's called the WCF per-session" lock. In any case, a lock is taken. I'm assuming that when ServiceB's GetDataB( ) is attempting to return the value, what's it's really done is flipped around the channel and is trying to call a receive method on ServiceA, which needs the lock and deadlocks.

in every question I've read so far where people ask "is it okay to chain one service from another", every (stupid) answer has been, "sure! go ahead! nothing to it! services don't know if they're being called by normal .NET code or if they're being called from another service!". Annoying. There certainly "is something to it", even if it's a little tiny bit trivial.

I've read that I can't mark methods that return values as "one way". And marking them as "re-entrant"... should that work or not? I know if in my service, I call a callback, the lock is silently released. What if I call another service? is WCF smart enough to release the lock in that case as well?

I could go mark my services as "multiple" instead of "re-entrant", but I'm wondering (a) is the deadlock really from what I think it is, or could it be from something else, and (b) is there a better way around this?

lokusking
  • 7,396
  • 13
  • 38
  • 57
eric frazer
  • 1,492
  • 1
  • 12
  • 19
  • The "stupid" responses you have seen are so, because simply chaining service invocations is (normally) really just that "stupidly" simple. There must be something else to your particular case. Have you enabled / looked at [WCF trace](http://stackoverflow.com/a/4271597/21567)? You might want to provide more (technical) information on your setup (code, config, etc.). – Christian.K Jul 09 '16 at 07:11
  • Of course you can chain service calls from other services, I've done it many, many times. Perhaps if you could host your code on Github (or atleast recreate it with a demo app that recreates the issue) or something, it might give people a better context of what you are working with to assist you. It seems that you might have reached a special snowflake scenario or maybe something has been misconfigured, who knows. – Dandré Jul 09 '16 at 07:18
  • so... the deadlock "shouldn't" happen? If the stack pointer is currently within ServiceA's GetDataA( ) method, and I call out to ServiceB's GetDataB( ) method, and GetDataB( ) executes and tries to call me back with the resultant data, it shouldn't be deadlocking by trying to take the receive lock back in ServiceA? I wrote a test app to demonstrate this, one with two services, and i'm surprised to NOT see it deadlock but I can't figure out why it's not deadlocking. Shouldn't it be? When B tries to return the data to A, shouldn't it try to obtain a lock and fail? – eric frazer Jul 09 '16 at 07:37
  • Indeed, if you got a deadlock it might be because of something else. WCF doesn't operate on the same thread as where the call originated from because then you would have seen deadlocks. If you see locks happening it might be because of an explicit lock that you've placed on your service or something buggy with the session bindings in WCF. I'm no expert but I haven't experienced any deadlocking from WCF before. – Dandré Jul 09 '16 at 07:49
  • chapter 8, concurrency, Programming WCF Services... he writes "If you call a proxy's method X, and on the server side, you want to call a callback, in order for it to not deadlock, since it will try to obtain the WCF Session Instance Lock, the Session's Concurrency Mode must be set to re-entrant, or the callback set to "one way"" That lock. If I am in A's method X, and I call B's method Y, which returns data, which is the same thing as calling me back, won't it try to take the Session's Instance Lock? – eric frazer Jul 09 '16 at 08:32
  • The WCF book is talking about callbacks, which is a particular feature of WCF. That has nothing to do with the general flow of chained requests. When your method GetDataB() is done, there is no need for WCF to hold any locks because (simplified) no user code is active any more. The result data itself is just marshalled bytes which require no locking whatsoever. I think @Dandré's suggestion to show your code, or better yet, a self contained example that exhibits the behavior you see is the only thing that will easily help clear up your confusion/issues. – Christian.K Jul 09 '16 at 09:00
  • You can pause the debugger and look at the stacks to get clues at what id deadlocking. Post the locked up stacks. Further, any WCF service that is not perinstance and concurrent is a smell. WCF services should be stateless. Further, cyclic dependencies between services often imply architectural problems. Could be a layering violation, or the services are too small and should be merged. Hope this helps. – usr Jul 09 '16 at 10:26
  • ericfrazer I wasn't aware that you tried to do callbacks, I was under the impression that you only made 1 directional calls, so I misunderstood you. I heard from a colleague that duplex channels are brittled with problems (he had lots of issues with it) and that Microsoft actually recommends against it, unfortunately I don't have the URLs to refer to, but I'm sure that you can see that you may need to avoid it. I agree with @usr it does seem like a design smell if you have a service call a service which calls the original service through the duplex channel. – Dandré Jul 09 '16 at 13:10
  • Yes, duplex channels are a smell too. This sounds like an OOP design forced on an SOA architecture. This does not have to be bad but is suspicious. – usr Jul 09 '16 at 13:17
  • @usr aren't duplex channels the way channel callbacks happen in WCF? – Dandré Jul 09 '16 at 13:25
  • Yes, callbacks are bad. They don't work well in practice and they cause the cyclic dependency problem. They are not fundamentally bad but they tend to not be the best choice. – usr Jul 09 '16 at 13:31
  • I work @ Microsoft, and work down the hall from one of the guys who helped write WCF. (Unfortunately, he left the group in it's infancy and isn't knee-deep in it any longer). He told me he thought maybe that calling a method that returned data was implemented as a (hidden) duplex channel, and that the same rules applied for using callbacks applied for calling methods that returned data. I think after experimenting that this is not true, since I am not seeing the deadlock happen in my litte test app. Like I said, in my little test app, I'm NOT seeing the deadlock happen, and am surprised. – eric frazer Jul 09 '16 at 14:44
  • I'm seeing a little bit of dis-information here. WCF services that are per-session certainly CAN have state, and WCF is written to allow that. It depends on your use case. In my case, both my services are written on top of TCP duplex channels, and one is on one machine, and one service is on a different machine. And no, duplex channels are not riddled with problems, but they're full of rules that allow problems to arise if not used correctly. What I'm actually writing is a video server. Node A is the master, node B is a worker node. B occasionally updates A with it's progress via a callback... – eric frazer Jul 09 '16 at 14:50
  • So then using something with lots of "red tape" might not be the best way to solve this problem, wouldn't you agree? Wouldn't it be better to have a WCF Operation endpoint exposed for Node B to call on Node A, rather than using the duplex channel? Unless you have some dependency that you don't want / can't introduce, it is an option that would alleviate the duplex channel dealock scenarios. Otherwise, another way to solve this would be to use an asynchronous messaging platform, like a publish/subscribe service or message queue system. – Dandré Jul 09 '16 at 14:59
  • I would have solved this by making A initiate the operation first in a quick call. A can then poll B for status, possibly using long polling which WCF supports. It is true that duplex channels and callbacks are not "buggy". They work as advertised. It's just usually bad architecture because it causes more labor than a simpler solution would have caused. – usr Jul 09 '16 at 15:05
  • but this is not what I was asking about in the original question, it was about if there is a deadlock problem when, while inside of a method of A, if a call to B that returns data will deadlock, and if not, why not. I tihnk I figured it out. When A calls B, A is waiting for B's reply, regardless if it returns data or not. B's reply is sent back on the channel w/o requesting the session lock. When A calls B and B tries to invoke a callback on A, that callback is out of band, and WILL request the session lock, causing a deadlock. – eric frazer Jul 09 '16 at 15:12
  • i agree an async message pattern would be best, and am leaning towards that. I am just trying to figure out how the channels actually transmit and receive data with regards to the session lock, and am asking questions about it, so others will know too. it's mysterious, all this one-way, async, concurrency, and locking stuff. I honestly thought A couldn't call B and get data back if A was stuck in the middle of a method. Wrong! :-) – eric frazer Jul 09 '16 at 15:15
  • Indeed, these discussions can get side-tracked! I'm glad you got to a solution. Would you mind posting the answer explaining what you found and how you fixed it? :-) – Dandré Jul 09 '16 at 15:24
  • 1
    I will certainly post the answer. I got a reference to the group that writes/maintains WCF internal to MS. i'm going to ask them for some insight then post the answer when I get it. – eric frazer Jul 09 '16 at 15:26

1 Answers1

1

Figured it out. And I must be honest, I started learning WCF by reading MSDN, but it droned on and on, so I moved on to some WCF online books (By Lowy & others). I should have kept reading MSDN.

I got this answer from one of the devs on the WCF team:

"The lock isn’t happening at the transport layer, it’s happening at the instance context layer. When a client in service A makes an outgoing request-reply call to service B, the reply isn’t going back to service A, it’s going back to the client for service B that service A is using."

That makes sense. But in addition to that information, I also learned this:

"If you use WCF services to call out of any method (call out = callback or another WCF service), and you're marked as "reenttrant", WCF will free the InstanceLock".

(Thus, the deadlock I'm seeing above is (probably) my own fault...)

I found this tidbit here:

https://channel9.msdn.com/Forums/TechOff/254330-WCF-ConcurrencyModeReentrant-Confusion

And that led me to this valuable article here:

https://msdn.microsoft.com/en-us/library/aa395214(v=vs.110).aspx

Quote: In case of Reentrant mode, the InstanceContext is unlocked just before the service makes an OUTGOING CALL thereby allowing the subsequent call, (which can be reentrant as demonstrated in the sample) to get the lock next time it comes in to the service.

eric frazer
  • 1,492
  • 1
  • 12
  • 19