4

This question is asking for advice as well as assistance with some code.

I currently am learning Python with 3.4 I have built a basic network checking tool, i import items from a text file and for each of them i want python to check dns (using pydns), ping the ip (using subprocess to call OS native ping).

Currently i am checking 5000 to 9000 thousand IP address and its taking a number of hours, approx 4 to return all the results.

I am wondering if i can use multiprocessing or threading to speed this up but still the return the output to a list so that the row can be written to a csv file at the very end of the script in bulk.

I am new to python so please tell me if i have overlooked something i should of also.

Main code http://pastebin.com/ZS23XrdE

Class http://pastebin.com/kh65hYhG

clonerworks
  • 641
  • 1
  • 6
  • 16
  • multiprocessing is probably a good bet, using a joinable queue to get the data back. But that might be no better than subprocess, depending on how you are running it. Are you waiting for each to complete? – cdarke Dec 12 '14 at 08:22
  • I have updated the main post with the current code i am using that is taking a long time. – clonerworks Dec 12 '14 at 09:02

2 Answers2

1

You could use multiple threads to run child processes (ping in your case) and collect their output but it is not necessary. Here's a code example how to make multiple http requests using a thread pool. Here's code that uses concurrent.futures to make dns requests concurrently.

You don't need multiple threads/process to check 5000-9000 IPs (DNS, ICMP). You could use gevent, twisted, asyncio to make network connections in the same process.

Community
  • 1
  • 1
jfs
  • 399,953
  • 195
  • 994
  • 1,670
0

As most of the work seems IO based, you can easily rely on Threads.

Take a look at the Executor.map() function in cocurrent.futures: https://docs.python.org/3/library/concurrent.futures.html

You can pass the list of IPs and the function you want to run against each element, the returned value, virtually, is the list of results of the given function.

In your specific case you can wrap the two worker's methods (check_dns_ip and os_ping) in a single one and pass it to the ThreadPoolExecutor.map function.

noxdafox
  • 14,439
  • 4
  • 33
  • 45