-1

I'm writing a server which accepts TCP connections and reads arbitrary-length lines of data terminated by a \n.

For this the semantics of socket.makefile(...).readline() does exactly what I need it to:

f = socket.makefile(mode="rb")

while True:
    data = f.readline()
    print(data.decode('utf-8'))
    if not data:
        break

However, there's an open issue on CPython alleging a performance issue:

https://github.com/python/cpython/issues/62529

What is the fast way for me read lines from the TCP socket that will work on both Linux and Windows?

Michael Ruth
  • 2,938
  • 1
  • 20
  • 27
Joseph Carrie
  • 1
  • 2
  • 2
    The issue mentions the faster way: `os.fdopen()`. – Barmar Aug 15 '23 at 20:32
  • 5
    This issue is pretty old (10 years) - have you run into any performance problems in practice? – JRiggles Aug 15 '23 at 20:32
  • 2
    That issue is about a performance degradation in `socket.makefile` between Python 2.7 and 3.2. It doesn't say that `socket.makefile` is prohibitively slow, it just says that it got a bit slower between 2.7 and 3.2. If it does exactly what you want, and you haven't actually measured a performance issue, I don't see why you wouldn't use it. – Brian61354270 Aug 15 '23 at 20:49
  • Do you need to read `\n` separated lines at all? It seems like this code separates the input by the `\n`, then uses `print()` to add the `\n` characters back in. Seems like you could use `socket.recv()` and write the bytes to stdout without a `\n`. – Nick ODell Aug 15 '23 at 20:53
  • @NickODell the code above is a highly simplified example – Joseph Carrie Aug 15 '23 at 21:06
  • @JRiggles I want to know up front what's the fastest thing to do, I want to learn – Joseph Carrie Aug 15 '23 at 21:07
  • @Barmar will that work on Windows? If it does, is there anything faster? – Joseph Carrie Aug 15 '23 at 21:08
  • Unfortunately, no. [`socket.fileno`](https://docs.python.org/3/library/socket.html#socket.socket.fileno) says "Under Windows the small integer returned by this method cannot be used where a file descriptor can be used (such as os.fdopen())" – Barmar Aug 15 '23 at 21:11
  • 3
    If you want to learn then run your own experiments and [profile them](https://stackoverflow.com/questions/582336/how-do-i-profile-a-python-script). Otherwise it doesn't matter what the "fastest way" is as long as your way is fast enough. – Woodford Aug 15 '23 at 21:47
  • _someone_ must know what the best/fastest way of doing this is though – Joseph Carrie Aug 16 '23 at 08:28
  • Rarely if ever is there a "best" way in programming. There may well be a way that "performs well enough in most typical situations", but that's about all you'll get. Everything has trade-offs and caveats. It's almost never worth it to get caught up in premature optimization when your time would be better spent getting things working. Like @Woodford said: if you want data, the best thing to do is try things out and see what works best *for your immediate needs*! It's also worth remembering that just because something fits a need in one project, it may well *not* fit a similar need elsewhere. – JRiggles Aug 16 '23 at 12:05

0 Answers0