Misuse of yield

Question

I'm making a SocketServer that will need to be able to handle a lot of commands. So to keep my RequestHandler from becoming too long it will call different functions depening on the command. My dilemma is how to make it send info back to the client.

Currently I'm making the functions "yield" everything it wants to send back to the client. But I'm thinking it's probably not the pythonic way.

# RequestHandler
func = __commands__.get(command, unkown_command)
for message in func():
    self.send(message)


# example_func
def example():
    yield 'ip: {}'.format(ip)
    yield 'count: {}'.format(count)

    . . .

    for ping in pinger(ip,count):
        yield ping

Is this an ugly use of yield? The only alterative I can think of is if when the RequestHandler calls the function is passes itself as an argument

func(self)

and then in the function

def example(handler):
    . . .
    handler.send('ip: {}'.format(ip))

But this way doesn't feel much better.

You could also use a lambda. Should make it easier for testing. — Sorin, Jul 21 '15 at 15:28
@Sorin Do you mean something like: func(lambda x: self.send(x))? — Lufftre, Jul 21 '15 at 15:33
Both solutions are technically correct, both are equally testable... The first one totally decouples the functions from the handler (which is fine if none of your function ever have to know anything from the handler), the second one is the most obvious for a Python newbie - that's how you'd do it in a language without generators. — bruno desthuilliers, Jul 21 '15 at 15:33
@Sorin I fail to see where would you use a lambda here and how it would make the testing easier... — bruno desthuilliers, Jul 21 '15 at 15:34
@brunodesthuilliers It's another option, beside the two mentioned already. They are all correct and all can be tested. Passing an object with send is an extra requirement that's not really needed so it makes it a bit more verbouse to test (you need to also create a dummy object). — Sorin, Jul 21 '15 at 15:43
You're making a coroutine. There's lots to read about that, but basically this is totally fine, although not super super common. — Marcin, Jul 21 '15 at 18:31

loopbackbee · Answer 1 · 2015-07-22T14:11:33.703

2

def example():
    yield 'ip: {}'.format(ip)
    yield 'count: {}'.format(count)

What strikes me as strange in this solution is not the use of yield itself (which can be perfectly valid) but the fact that you're losing a lot of information by turning your data into strings prematurely.

In particular, for this kind of data, simply returning a dictionary and handling the sending in the caller seems more readable:

def example():
    return {'ip': ip, 'count': count}

This also helps you separate content and presentation, which might be useful if you want, for example, to return data encoded in XML but later switch to JSON.

If you want to yield intermediate data, another possibility is using tuples: yield ('ip', ip). In this way you keep the original data and can start processing the values immediately outside the function

edited Jul 22 '15 at 14:11

answered Jul 21 '15 at 18:29

loopbackbee

21,962
10
62
97

I don't want to keep the client waiting for a response. As soon as the information is retrieved the client must receive it. – Lufftre Jul 22 '15 at 10:31
@Lufftre You can use tuples then: `yield ('ip', ip)`. But note that this makes things harder and, unless getting the ip takes a lot of time or the data is really big, it will probably degrade performance considerably and have no discernible advantage (see [premature optimization](http://stackoverflow.com/questions/385506/when-is-optimisation-premature)) – loopbackbee Jul 22 '15 at 11:20
Sorry my example is pretty bad, but yes sometimes it takes a lot of time to get the data. I like the idea of returning tuples, will look into that. – Lufftre Jul 22 '15 at 11:35

score 0 · Answer 2 · answered Sep 13 '21 at 14:42

I do the same as you with yield. The reason for this is simple:

With yield the main loop can easily handle the case that sending data to one socket will block. Each socket gets a buffer for outgoing data that you fill with the yield. The main loop tries to send as much of that as possible to the socket and when it blocks it records how far it got in the buffer and waits for the socket to be ready for more. When the buffer is empty is runs next(func) to get the next chunk of data.

I don't see how you would do that with handler.send('ip: {}'.format(ip)). When that socket blocks you are stuck. You can't pause that send and handle other sockets easily.

Now for this to be useful there are some assumptions:

the data each yield sends is considerable and you don't want to generate all of it into one massive buffer ahead of time
generating the data for each yield takes time and you want to already send the finished parts
you want to use reply = yield data waiting for the peer to respond to the data in some way. Yes, you can make this a back and forth. next(func) becomes func.send(reply).

Any of these is a good reason to go the yield way or coroutines in general. The alternative seems to be to use one thread per socket.

Note: the func can also call other generators using yield from. Makes it easy to split a large problem into smaller handlers and to share common parts.

Misuse of yield

2 Answers2