1

I have some python script with more than 3k outgoing socket connections, based on asyncore lib. I can't use select(..) due to connections limit (1024), but poll(..) not working properly too:

asyncore.loop(use_poll=True)

With this invocation my app ignores any socket events. Note that select() on less than 1024 sockets works fine.

Where is my problem?

Pavel Kirienko
  • 1,162
  • 1
  • 15
  • 31
  • 1
    the title of your post suggests, you got an exception. how about adding the stacktrace to your question? – Roshan Mathews Nov 03 '11 at 15:42
  • Okay, stacktrace is here: http://pastebin.com/WQEKtHHY But it not related to my question. – Pavel Kirienko Nov 03 '11 at 15:59
  • This sounds like an OS question. On any UNIX-like OS, you can increase the maximum number of open sockets with a config option and reboot. On Windows, it should also be possible, at least on the server versions. – Michael Dillon Nov 03 '11 at 20:22
  • 2
    No, this is not a resource limitation problem - select() simply unable to handle more than 1024 sockets (http://stackoverflow.com/questions/5357445/select-max-sockets). My problem is "my app ignores any socket events with poll()". – Pavel Kirienko Nov 04 '11 at 11:43

1 Answers1

0

First, a minor correction. use_poll=True does not cause asyncore to use epoll(2). It causes it to use poll(2) (moreover, the way asyncore works, there is little reason to try to use epoll(2), because the Python-level overhead of asyncore overwhelms any cost of the socket event notification API being used).

Second, a major shortcoming of asyncore is the degree to which it exposes you to platform-specific quirks. For a library that handles more of the differences between select(2) and poll(2) for you, and which actually supports epoll(2), and which is better in a number of other ways too, check out Twisted.

If you can add more details to your question, perhaps the specific problem you're encountering on asyncore can be discovered, but based on the information available now, there's no way to be sure. The very basic features of asyncore's poll(2) support do work - ie, it can deliver read, write, and close notifications in the trivial case. How does your case differ from the trivial case?

Community
  • 1
  • 1
Jean-Paul Calderone
  • 47,755
  • 6
  • 94
  • 122
  • "use_poll=True does not cause asyncore to use epoll" - yes, sorry, I did mean poll(). I simply need to handle about 3000 sockets by single script, performance doesn't matter. Here's my code: http://pastebin.com/ttu7LJJv. I would happy to see simple working example of asyncore with poll(). Thanks. – Pavel Kirienko Dec 08 '11 at 16:21
  • And, strictly speaking, the problem is no longer relevant, but I still want to understand it. – Pavel Kirienko Dec 08 '11 at 16:24
  • Does the problem appear when you use spin_once, spin, or spin_all? Do you have a simple server that this can connect to (I can probably build one but it'd be easier not to have to :). – Jean-Paul Calderone Dec 08 '11 at 16:46
  • Sorry for taking so long to answer. I found the cause of my problem (it took just 10 minutes of experimentation), look: http://pastebin.com/yYawd9cy, especially at last two lines of code. As you can see, asyncore's poll() doesn't work with explicitly specified map. Original version is here: http://docs.python.org/release/2.5.2/lib/asyncore-example.html. – Pavel Kirienko Dec 13 '11 at 07:11
  • Well, I understood everything. http://pastebin.com/TtE9wSSm, look at asyncore.dispatcher.__init__(self, map=mymap). My problem was in incorrect initialization of asyncore's map (this point is poorly documented). Thanks. – Pavel Kirienko Dec 13 '11 at 07:33