3

We have a gen_server process that manages the pool of passive sockets on the client side by creating them and borrowing them for other processes. Any other process can borrow a socket, sends a request to the server using the socket, gets a reply through gen_tcp:recv, and then releases the socket to the gen_server socket pool process.

The socket pool process monitors all processes that borrow the sockets. If any of the borrowed process is down, it gets a down signal from it:

handle_info({'DOWN', Ref, process, _Pid, _Reason}, State) ->

In this case we would like to drain the borrowed socket, and reuse it by putting back into the pool. The problem is that while trying to drain a socket using gen_tcp:recv(Socket, 0, 0), we get inet ealready error message, meaning that recv operation is in progress.

So the question is how to interrupt previous recv, successfully drain a socket, and reuse for other processes.

Thanks.

Mayya Sharipova
  • 405
  • 4
  • 11
  • It's interesting that gen_tcp:recv behaves that way, I never noticed before, but then I would always use active sockets. I wonder if this is sensible, or for that matter intended behaviour. If the process blocking on gen_tcp recv dies I'd really expect/want the read to be interupted, but instead it sticks until it receives enough data that the gen_tcp:recv call would have been satisfied. Could you consider using active sockets? I think it would be a lot safer and let you do everything OTP style. – Michael Dec 04 '15 at 10:35
  • It would be interesting to hear a comment from someone familiar with the inet driver (inet_drv.c) really? – Michael Dec 04 '15 at 10:35
  • Thanks @Michael, good suggestion, will try active sockets as well – Mayya Sharipova Dec 04 '15 at 17:20

1 Answers1

2

One more level of indirection will greatly simplify the situation.

Instead of passing sockets to processes that need to use them, have each socket controlled by a separate process that owns it and represents the socket within the system. Route Erlang-side messages to and from sockets as necessary to implement the "borrowing" of sockets (even more flexibly, pass the socket controller a callback module that speaks a given protocol, so as soon as data comes over the network it is interpreted as Erlang messages internally).

If this is done you will not lose control of sockets or have them in indeterminate states -- they will instead be held by a single, owning process the entire time. Instead of having the route-manager/pool-manager process receive the 'DOWN' messages, have the socket controllers monitor its current using process. When a 'DOWN' is received then you can change state according to whatever is necessary.

You can catch yourself in some weird situations passing open files descriptors, socket and other types of ports around among sockets that aren't designated as the owner of them. Passing ports and sockets around also becomes a problem if you need to scale a program across several nodes (suddenly you have to care where things are passed and what node they are on, etc.).

zxq9
  • 13,020
  • 1
  • 43
  • 60
  • Technically, I'm not sure I see how this solves the OPs problem. Any process that 'owns' the socket can die, and whilst it may be more likely for arbitrary borrowers to die than a simple well crafted socket intermediary, it could still happen... and if it does the socket will get stuck still with no other process able to take it over... – Michael Dec 04 '15 at 10:24
  • Also, if you're going to turn the data into messages, wouldn't using active sockets be better, simpler, and also not prone to the recv blocking problem? – Michael Dec 04 '15 at 10:26
  • Thanks zxq9 for the suggestion & explanation, will try to implement this approach. @Michael - you are right, using active sockets could be an alternative approach, going to try it as well, just have heard it is slower than passive sockets, need to do some comparison here. – Mayya Sharipova Dec 04 '15 at 17:19
  • Sockets in active mode are equivalent to a process, from another process' perspective. Wrapping a socket in another process just increases copying which leads to worse performance. – Adam Lindberg Dec 08 '15 at 13:29
  • @AdamLindberg Ordinary processes do not have *controlling processes*. Pooling sockets like this convolutes the semantics of the code in a way that having a *genuine* process that represents the socket *as its controller* does not. Performance is not everything, especially when you run into the sort of coding tangle the OP indicates. Also: You tend to get a greater performance gain from writing a controlling `proc_lib` process in pure Erlang than trying to handle them with `gen_server` -- so if response latency to a socket event is the concern, that's the bigger gain. – zxq9 Dec 08 '15 at 13:35
  • The controlling process is just the owner of the socket, it doesn't mean that only that process should read from and write to the socket. My point is more or less that you shouldn't wrap sockets in processes unless you have to. A more lightweight solution is to have a socket manager and let other processes have access to the sockets directly. – Adam Lindberg Dec 21 '15 at 10:33
  • @AdamLindberg I usually (but not always) argue to take the performance hit over letting semantic drift creep into code. This is one of those cases where "yes, you *can* do that, but that doesn't mean you *should*". But like anything else in coding, which approach to take depends on a big pile of context that isn't a part of this post. – zxq9 Dec 21 '15 at 10:46