How non-blocking web server works?

Question

I'm trying to understand the idea of non-blocking web server and it seems like there is something I miss.

I can understand there are several reasons for "block" web request(psuedocode):

1. CPU bound

   string on_request(arg)
   {
   DO_SOME_HEAVY_CPU_CALC
   return "done";
   }

2. IO bound

   string on_request(arg)
   {
   DO_A_CALL_TO_EXTERNAL_RESOURCE_SUCH_AS_WEB_IO
   return "done";
   }

3. sleep

   string on_request(arg)
   {
   sleep(VERY_VERY_LONG_TIME);
   return "done";
   }

   • are all the three can benefit from non-blocking server?
   • how the situation that do benefit from the non-blocking web server really do that? I mean, when looking at the Tornado server documentation, it seems like it "free" the thread. I know that a thread can be put to sleep and wait for a signal from the operation system (at least in Linux), is this the meaning of "freeing" the thread? is this some higher level implementation? something that actually create a new thread that is waiting for new request instead of the "sleeping" one?
   • Am I missing something here?

Thanks

Answer 1

Basically the way the non-blocking sockets I/O work is by using polling and the state machine. So your scheme for many connections would be something like that:

• Create many sockets and make them nonblocking
• Switch the state of them to "connect"
• Initiate the connect operation on each of them
• Poll all of them until some events fire up
• Process the fired up events (connection established or connection failed)
• Switch the state those established to "sending"
• Prepare the Web request in a buffer
• Poll "sending" sockets for WRITE operation
• send the data for those who got the WRITE event set
• For those which have all the data sent, switch the state to "receiving"
• Poll "receiving" sockets for READ operation
• For those which have the READ event set, perform read and process the read data according to the protocol
• Repeat if the protocol is bidirectional, or close the socket if it is not

Of course, at each stage you need to handle errors, and that the state of each socket is different (one may be connecting while another may be already reading).

Regarding polling I have posted an article about how different polling methods work here: http://www.ulduzsoft.com/2014/01/select-poll-epoll-practical-difference-for-system-architects/ - I suggest you check it.

Answer 2

To benefit from a non-blocking server, your code must also be non-blocking - you can't just run blocking code on a non-blocking server and expect better performance. For example, you must remove all calls to sleep() and replace them with non-blocking equivalents like IOLoop.add_timeout (which in turn involves restructuring your code to use callbacks or coroutines).

Answer 3

How To Use Linux epoll with Python http://scotdoyle.com/python-epoll-howto.html may give you some points about this topic.