I have a script that uses easysnmp get and bulkwalk modules to gather a list of all the IP addresses on a device and add them into a dictionary for further processing. The problem happens with specific IP addresses and the issue does not seem to be with the end hosts themselves.
From a fresh start of a python shell, I can do a bulkwalk on one of the problem IPs:
>>> from easysnmp import snmp_bulkwalk
>>> snmp_ips = snmp_bulkwalk('ipAdEntIfIndex', hostname='x.x.x.80', version=3, security_username=snmp_user, security_level='auth_with_privacy', auth_protocol='SHA', auth_password=snmp_auth, privacy_protocol='AES128', privacy_password=snmp_priv)
>>> snmp_ips
[<SNMPVariable value='4010' (oid='ipAdEntIfIndex', oid_index='x.x.x.179', snmp_type='INTEGER')>, <SNMPVariable value='4011' (oid='ipAdEntIfIndex', oid_index='x.x.x.186', snmp_type='INTEGER')>, <SNMPVariable value='20567' (oid='ipAdEntIfIndex', oid_index='x.x.x.80', snmp_type='INTEGER')>, <SNMPVariable value='808' (oid='ipAdEntIfIndex', oid_index='x.x.x.133', snmp_type='INTEGER')>]
If I run my function against just this single node, it works. Here is the function:
def get_ints():
addrs = {}
for h in nodes['results']:
# Update all nodes in the dictionary to their FQDN
if 'foo.com' not in h['SysName']:
h['SysName'] = h['SysName'] + '.foo.com'
# Fix random hostname issues
if '-re' in h['SysName']:
# This is probably way more complicated than it needs to be. Split the hostname at '-' and combine the two results
# starting at index 3 of the second half of the split.
h['SysName'] = h['SysName'].split('-'[::])[0] + h['SysName'].split('-'[::])[1][3:]
host = h['SysName']
# Use SNMP to get IP and interface information because screen scraping is slow and painful to reformat.
# This walks the IF-MIB table and gets the IPs assigned to each interface.
print(f'Getting interface info from {host}')
try:
snmp_ips = snmp_bulkwalk('ipAdEntIfIndex', hostname=host, version=3, security_username=snmp_user, security_level='auth_with_privacy', auth_protocol='SHA', auth_password=snmp_auth, privacy_protocol='AES128', privacy_password=snmp_priv)
except:
print(f'Could not get data from {host}')
# Iterate through SNMP GET objects and update the addrs table with: { IP : { PTR record : RR value }}
for x in snmp_ips:
# We only want to process IPs that fall within specific network ranges
for n in my_nets:
if ip_address(x.oid_index) in n:
try:
addrs[x.oid_index] = gen_rev(x.oid_index, snmp_get('ifName' + '.' + x.value, hostname=host, version=3, security_username=snmp_user, security_level='auth_with_privacy', auth_protocol='SHA', auth_password=snmp_auth, privacy_protocol='AES128', privacy_password=snmp_priv).value.lower(), host)
except:
pass
return addrs
Here's the function being run against the single host in the dictionary:
>>> nodes['results'] = [ {'NodeID': 571, 'SysName': 'host80.foo.com', 'IPAddress': 'x.x.x.80' } ]
>>> my_addrs = get_ints()
>>> Getting interface info from host80.foo.com
>>> my_addrs
[<SNMPVariable value='4010' (oid='ipAdEntIfIndex', oid_index='x.x.x.179', snmp_type='INTEGER')>, <SNMPVariable value='4011' (oid='ipAdEntIfIndex', oid_index='x.x.x.186', snmp_type='INTEGER')>, <SNMPVariable value='20567' (oid='ipAdEntIfIndex', oid_index='x.x.x.80', snmp_type='INTEGER')>, <SNMPVariable value='808' (oid='ipAdEntIfIndex', oid_index='x.x.x.133', snmp_type='INTEGER')>]
Here's a full set of nodes in the dictionary and when the function is run against the full set, 'host80.foo.com' fails:
>>> nodes['results'] = [{'NodeID': 14, 'SysName': 'host13.foo.com', 'IPAddress': 'x.x.x.13'}, {'NodeID': 15, 'SysName': 'host14.foo.com', 'IPAddress': 'x.x.x.14'}, {'NodeID': 17, 'SysName': 'hhost16.foo.com', 'IPAddress': 'x.x.x.16'}, {'NodeID': 18, 'SysName': 'host17.foo.com', 'IPAddress': 'x.x.x.17'}, {'NodeID': 571, 'SysName': 'host80.foo.com', 'IPAddress': 'x.x.x.80'}, {'NodeID': 20, 'SysName': 'host19.foo.com', 'IPAddress': 'x.x.x.19'}]
>>> my_addrs = get_ints()
Getting interface info from host13.foo.com
Getting interface info from host14.foo.com
Getting interface info from host16.foo.com
Getting interface info from host17.foo.com
Getting interface info from host19.foo.com
Getting interface info from host80.foo.com
Could not get data from host80.foo.com
...
Now if I try to run just the bulkwalk request on its own, it no longer works:
>>> snmp_bulkwalk('ipAdEntIfIndex', hostname='x.x.x.80',version=3, security_username=snmp_user, security_level='auth_with_privacy', auth_protocol='SHA', auth_password=snmp_auth, privacy_protocol='AES128', privacy_password=snmp_priv)
easysnmp.exceptions.EasySNMPTimeoutError: timed out while connecting to remote host
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/logging/__init__.py", line 1293, in debug
if self.isEnabledFor(DEBUG):
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/logging/__init__.py", line 1548, in isEnabledFor
return level >= self.getEffectiveLevel()
SystemError: PyEval_EvalFrameEx returned a result with an error set
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/easysnmp/easy.py", line 147, in snmp_bulkwalk
return session.bulkwalk(oids, non_repeaters, max_repetitions)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/easysnmp/session.py", line 498, in bulkwalk
interface.bulkwalk(self, non_repeaters, max_repetitions, varlist)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/logging/__init__.py", line 1293, in debug
if self.isEnabledFor(DEBUG):
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/logging/__init__.py", line 1548, in isEnabledFor
return level >= self.getEffectiveLevel()
SystemError: PyEval_EvalFrameEx returned a result with an error set
If I exit out of the interpreter and do it all over again, it's always the same result. I can also run the standalone bulkrequest in another python shell and it works fine.
Packet captures at my firewall show the request being sent to the device, but instead of a get-response, the device sends a report response. There does not seem to be any difference at all between get requests and getbulkrequests between sessions that work and sessions that don't work. I'm willing to accept that the problem might be something to do with the firewall, but what, especially since the bulkrequest works fine in another shell at the same time? Also, why does it work on its own and not when the function is run against all the hosts in the table, every single time?
I've tried adding imported modules in one at a time to see if there is something happening on that front, but I don't ever have a failure until the function is run against the rest of the hosts in the dictionary and will not work at all afterwards in that same shell.
