Fastest pinging method via python?

Question

I'm looking for the fastest pinging method via python. I need to ping over 100,000 servers and my current procedure below takes approximately 85 minutes to complete. I've read small snippets about scapy, along with general ICMP and python ping. I need to know a definitive method, or at least a solid way to test, which is the fastest. I cannot test python - ping from work as it is not an approved package. I also tried a code snippet for scapy, but got an error:

OSError: Windows native L3 Raw sockets are only usable as administrator !
Install 'Winpcap/Npcap to workaround !

So I'm admittedly looking for code snippets I can test at home or ways around that error from more experienced persons

To prove I've tried, here are some related posts, as well as my current code

Current code:

import pandas as pd
import subprocess
import threading
raw_list = []
raw_list2 = []
def ping(host):
    raw_list.append(host+ ' '+ str((subprocess.run('ping -n 3 -w 800 '+host).returncode)))
with open(r"FILEPATH", "r") as server_list_file:
    hosts = server_list_file.read()
    hosts_list = hosts.split('\n')
num_threads = 100
num_threads2 = 10
num_threads3 = 1
number = 0
while number<len(hosts_list):
    print(number)
    if len(hosts_list)>number+num_threads:
        for i in range(num_threads):
            t = threading.Thread(target=ping, args=(hosts_list[number+i],))
            t.start()
        t.join()
        number = number + num_threads
    elif len(hosts_list)>(number+num_threads2):
        for i in range(num_threads2):
            t = threading.Thread(target=ping, args=(hosts_list[number+i],))
            t.start()
        t.join()
        number = number + num_threads2
    elif len(hosts_list)>(number+num_threads3-1):
        for i in range(num_threads3):
            t = threading.Thread(target=ping, args=(hosts_list[number+i],))
            t.start()
        t.join()
        number = number + num_threads3
    else:
        number = number+1
for x in range(len(raw_list)):
    if(raw_list[x][-1] == '0'):
        raw_list2.append(raw_list[x][0:-2])
to_csv_list = pd.DataFrame(raw_list2)
to_csv_list.to_csv('ServersCsv.csv', index = False, header = False)
to_csv_list.to_csv(r'ANOTHERFILEPATH', index = False, header = False) 
subprocess.call(r'C:\ProgramData\Anaconda3\python.exe "A_PROGRAM_THAT_INSERTS_INTO_SQL"')

This does exactly what I need, however, it does not do it quickly enough.

I've tried the very small snippet:

from scapy.all import *
packets = IP(dst=["www.google.com", "www.google.fr"])/ICMP()
results = sr(packets)

resulting in gaierror: [Errno 11001] getaddrinfo failed

I've also tried:

TIMEOUT = 2
conf.verb = 0
packet = IP("ASERVERNAME", ttl=20)/ICMP()
reply = sr1(packet, timeout=TIMEOUT)
if not (reply is None):
     print(reply.dst + "is online")
else:
     print("Timeout waiting for %s") % packet[IP].dst

resulting in:

OSError: Windows native L3 Raw sockets are only usable as administrator !
Install Winpcap/Npcap to workaround !

A few links I looked at but could not garner a solid answer from:

Ping a site in Python?

Fastest way to ping a host in python?

Do you really need to ping 100,000 servers? Starting a new process for each one seems rather excessive - try doing it internally to your process. If you have large amounts of high-latency I/O, you should be looking at a threaded or async solution. — Mark Setchell, Feb 03 '20 at 21:08
1. Yes I do need to ping over 100,000 servers daily. I work for an excessively large company and this helps validate data on which servers are powered on and which have connectivity etc. 2. Yes, starting a new process for each one does seem excessive. Im aware my code probably isnt anywhere near optimized, and that is why Im asking for help :) — AlbinoRhino, Feb 03 '20 at 21:09
Been too long since I wrote networking code for me to remember the exact details, but I would write a C program with two threads: One thread would continually blast out ICMP echo requests to all of the servers that had not yet replied, and the other thread would continually listen for replies, and check them off as they arrived. ...Maybe some kind of throttling to avoid DOSing the local network...Maybe some smarts to back off from aggressively re-trying servers that really are down...Don't have enough Python knowledge to know if you could do the same in that language. — Solomon Slow, Feb 03 '20 at 21:48
This was my first multithreaded program ever, but I will look into this method — AlbinoRhino, Feb 03 '20 at 21:53
**why** must it be fast? And: if it *really* must be fast: don't use windows, dont use python. — wildplasser, Feb 04 '20 at 00:02
I don't really see any 'legal' usage that would require pinging 100k servers. Except maybe if you work for a domain name registrar or Cloudflare... in which case you wouldn't be using a python script :-) — Cukic0d, Feb 04 '20 at 00:05
I work for a major payment processor. we have over 60,000 virtual machines, plus their hosts, external things in various countries that need pinged. are you being serious @Cukic0d — AlbinoRhino, Feb 04 '20 at 20:36
@wildplasser it doesnt NEED absolute speed. We didnt do this last week even. but it would be nice to optimize things. I only know python and have only ever used windows. — AlbinoRhino, Feb 04 '20 at 20:37

Cukic0d · Answer 1 · 2020-02-04T00:00:29.410

This only solves the Python part. The comments are very right.

OSError: Windows native L3 Raw sockets are only usable as administrator ! Install Winpcap/Npcap to workaround !

I find this pretty damn explicit. If you follow's Scapy documentation for windows it says you need to install Npcap.https://nmap.org/npcap/

Other than that,

packets = IP(dst=["www.google.com", "www.google.fr"])/ICMP()
results = sr(packets)

Is likely the cleanest way to go. Works on my machine.. make sure you're using the latest development version from GitHub (unzip it and install it via python setup.py install).

If you are using the latest version, you might even want to turn on threaded=True in sr() to send and receive packets on two threads, as pointed out by the comments. You might also want to use prn and store=False to not store the answers (100k is a lot)

Fastest pinging method via python?

1 Answers1