Threaded Python script only seems to work as expected after signaling keyboard interrupt?

Question

Basically, I have a script that sniffs packets in one thread, and appends them to a list. A second thread also runs, and if the list is not empty, then it will perform calculations with the data. However, the sniffing thread only seems to work properly after hitting Ctrl + C. Before sending the keyboard interrupt, console output that I've produced for debugging is very slow and it seems to be missing packets. Afterwards hitting Ctrl + C it runs much faster and works as expected. Any ideas why this might be occurring? My code would look something similar to what is below.

Packet sniffer:

import socket, sys
from struct import *

def eth_addr(a): 
    #create a AF_PACKET type raw socket (thats basically packet level)
    #define ETH_P_ALL    0x0003          /* Every packet (be careful!!!) */
    try:
        s = socket.socket( socket.AF_PACKET , socket.SOCK_RAW , socket.ntohs(0x0003))
    except socket.error , msg:
        print 'Socket could not be created. Error Code : ' + str(msg[0]) + ' Message ' + msg[1]
        sys.exit()

def packet_sniff():     
    # receive a packet
    while True:
        packet = s.recvfrom(65565)

        #packet string from tuple
        packet = packet[0]

        #parse ethernet header
        eth_length = 14

        eth_header = packet[:eth_length]
        eth = unpack('!6s6sH' , eth_header)
        eth_protocol = socket.ntohs(eth[2])
        print 'Destination MAC : ' + eth_addr(packet[0:6]) + ' Source MAC : ' + eth_addr(packet[6:12]) + ' Protocol : ' + str(eth_protocol)

        #TCP protocol
        if protocol == 6 :
            t = iph_length + eth_length
            tcp_header = packet[t:t+20]

            #now unpack them
            tcph = unpack('!HHLLBBHHH' , tcp_header)

            source_port = tcph[0]
            dest_port = tcph[1]
            sequence = tcph[2]
            acknowledgement = tcph[3]
            doff_reserved = tcph[4]
            tcph_length = doff_reserved >> 4

            print 'Source Port : ' + str(source_port) + ' Dest Port : ' + str(dest_port) + ' Sequence Number : ' + str(sequence) + ' Acknowledgement : ' + str(acknowledgement) + ' TCP header length : ' + str(tcph_length)

            h_size = eth_length + iph_length + tcph_length * 4
            data_size = len(packet) - h_size

            #get data from the packet
            data = packet[h_size:]

            return data

Main would look something roughly like this:

def run_program():
    processThread1 = threading.Thread(target = self.sniff_data, args = [])   
    processThread2 = threading.Thread(target = self.process_data, args = [])
    processThread1.start()
    processThread2.start()

def sniff_data():
    global my_list
    while True:
        data = packet_sniff()
        my_list.append(data)

def process_data():
    global my_list
    while True:
        if len(my_list) != 0:
            # Do computation

my_list = []
run_program()

First impulse: use [`if __name__ == "__main__":`](https://stackoverflow.com/questions/419163/what-does-if-name-main-do) for your main thread, you may be creating more threads than you intend. — poompt, Jul 23 '18 at 15:39
@poompt thank you for your input, I should definitely get into that habit of using that more, unfortunately the problem still persists — Francis, Jul 23 '18 at 15:54
I'm just trying to understand your code here. In your first module, why is the first try statement indented? For `if protocol == 6 :`, same question. — poompt, Jul 23 '18 at 18:53
Also, depending on what `process_data` is doing it could be locking your whole process if it just runs as fast as it can. If it could potentially eat the whole CPU, you'd typically put a `time.sleep()` in the loop or use a [process](https://docs.python.org/2/library/multiprocessing.html) instead. — poompt, Jul 23 '18 at 18:58
@poompt My apologies that was just a typo, it is edited now. Interesting, that sounds very possible. I’ll experiment with it. However my follow up question would be is that any reason that the keyboard interrupt would free the CPU even though the data is still being processed as specified? — Francis, Jul 23 '18 at 19:04
No idea on if this order of events would happen, but what I'm thinking is along the lines of: Both threads start; sniff_data waits patiently for a packet; process_data continually checks my_list to see if it is empty. It does this as fast as the CPU allows; sniff_data is now not actually checking for a packet because my_list is eating the whole logical processor; you exit process_data, freeing up my_list long enough for sniff_data to put something in it; somehow process_data is up again but now my_list is not empty, so it doesn't eat the CPU. possibly an extra one spawned from main module. — poompt, Jul 23 '18 at 19:12
Specifically, `if len(my_list) != 0:` with no sleep run infinitely can and will lock the interpreter, preventing other threads from changing `my_list`. That needs to go. — poompt, Jul 23 '18 at 19:14
@poompt You're officially my hero, that bug was driving me crazy, and sleeping solved the issue, so it seems to be exactly as you mentioned. Thank you for your help. — Francis, Jul 23 '18 at 19:21

score 0 · Accepted Answer · answered Jul 23 '18 at 19:26

0

if len(my_list) != 0: with no sleep run infinitely can and will lock the interpreter, preventing other threads from changing my_list. That needs to go.

answered Jul 23 '18 at 19:26

poompt

248
1
8

Threaded Python script only seems to work as expected after signaling keyboard interrupt?

1 Answers1