1

I have time consuming SNMP walk task to perform which I am running as a background process using Popen command. How can I capture the output of this background task in a log file. In the below code, I am trying to do snampwalk on each IP in ip_list and logging all the results to abc.txt. However, I see the generated file abc.txt is empty.

Here is my sample code below - 

import subprocess
import sys

f = open('abc.txt', 'a+')

ip_list = ["192.163.1.104", "192.163.1.103", "192.163.1.101"]

for ip in ip_list:
    cmd = "snmpwalk.exe -t 1 -v2c -c public "
    cmd = cmd + ip
    print(cmd)
    p = subprocess.Popen(cmd, shell=True, stdout=f)
    p.wait()
f.close()

print("File output - " + open('abc.txt', 'r').read())

the sample output from the command can be something like this for each IP - 

sysDescr.0 = STRING: Software: Whistler Version 5.1 Service Pack 2 (Build 2600)
sysObjectID.0 = OID: win32
sysUpTimeInstance = Timeticks: (15535) 0:02:35.35
sysContact.0 = STRING: unknown
sysName.0 = STRING: UDLDEV
sysLocation.0 = STRING: unknown
sysServices.0 = INTEGER: 72
sysORID.4 = OID: snmpMPDCompliance

I have already tried Popen. But it does not logs output to a file if it is a time consuming background process. However, it works when I try to run background process like ls/dir. Any help is appreciated.

shivam gupta
  • 471
  • 5
  • 10
  • You can redirect the `Popen` output to a file if you wish, `Popen(..., stdout=..)`. – Torxed May 18 '20 at 16:55
  • import subprocess import sys f = open('abc.txt', 'w') cmd = "snmpwalk.exe -t 1 -v2c -c public 192.168.34.3" p = subprocess.Popen(cmd, shell=True, stdout=f) sys.stdout.flush() f.close() – shivam gupta May 18 '20 at 16:59
  • @Torxed, I have tried passing file object to stdout. But it does not write the out to the file – shivam gupta May 18 '20 at 17:01
  • You could also just read `handle.stdout.read()` every now and then? :) – Torxed May 18 '20 at 17:02
  • @Torxed I tried that too but stout.read does not print anything for a long background process. So, this does not solve the problem – shivam gupta May 18 '20 at 19:09
  • It does, but you probably used `.readline()` or something similar, and as I have no clue what your "broken" code looked like or what you've attempted, I can't help you either until that information is clear to us. – Torxed May 18 '20 at 20:32
  • @Torxed, I have updated my sample code in the problem above. Please have a look. Thanks! – shivam gupta May 19 '20 at 04:42
  • Definate duplicate of https://stackoverflow.com/questions/1606795/catching-stdout-in-realtime-from-subprocess – Torxed May 19 '20 at 06:52

1 Answers1

0

The main issue here is the expectation of what Popen does and how it works I assume. p.wait() here will wait for the process to finish before continuing, that is why ls for instance works but more time consuming tasks doesn't. And there's nothing flushing the output automatically until you call p.stdout.flush().

The way you've set it up is more meant to work for:

  1. Execute command
  2. Wait for exit
  3. Catch output

And then work with it. For your usecase, you'd better off using an alternative library or use the stdout=subprocess.PIPE and catch it yourself. Which would mean something along the lines of:

import subprocess
import sys

ip_list = ["192.163.1.104", "192.163.1.103", "192.163.1.101"]
with open('abc.txt', 'a+') as output:
    for ip in ip_list:
        print(cmd := f"snmpwalk.exe -t 1 -v2c -c public {ip}")
        process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) # Be wary of shell=True
        while process.poll() is None:
            for c in iter(lambda: process.stdout.read(1), ''):
                if c != '':
                    output.write(c)

with open('abc.txt', 'r') as log:
    print("File output: " + log.read())

The key things to take away here is process.poll() which checks if the process has finished, if not, we'll try to catch the output with process.stdout.read(1) to read one byte at a time. If you know there's new lines coming, you can switch those three lines to output.write(process.stdout.readline()) and you're all set.

Torxed
  • 22,866
  • 14
  • 82
  • 131
  • thanks for your explanation. But, this does not solves the issue. Nothing gets captured in the file. It is empty – shivam gupta May 19 '20 at 10:58
  • If that's the case, then there's truly no ouput from the command. Could you perhaps edit your question to give us example output from the command? – Torxed May 19 '20 at 11:27
  • I have added the sample output in the question for your reference. Also, when I am manually trying the command in cmd, it is working fine and I see the output on the console – shivam gupta May 19 '20 at 12:18
  • 1
    @shivamgupta I'm having a hard time following what doesn't work for you. Here's a detailed video of a SNMP server listening (bottom), a tail of the `abc.txt` file as I run the code above (middle window) and the code in this answer (top window): https://youtu.be/2uBrYWcmjrM - As you can see, the output to `abc.txt` is near instantaneous and I'm not quite sure what you mean by "nothing gets captured in the file" since it obviously does. I ran this on a Linux machine, because I couldn't find the windows binary of net-snmp. Perhaps you've compiled it strangely? output to `stderr`? – Torxed May 19 '20 at 13:04
  • I am following you. But nothing gets printed to the log file. Can it be a windows issue, because I am on windows machine – shivam gupta May 19 '20 at 13:48
  • @shivamgupta Doubtful, the pipeline in the library user layer is identical as far as I know. Either it's some form of buffer issue with the binary that's compiled or it's an issue I've never ever heard of before. Is it possible for you to direct me to how and where you found the windows binary? Perhaps I can try it out as well. – Torxed May 19 '20 at 14:40
  • I am using "Get-Content abc.txt -Wait -Tail 2 " windows command for monitoring the log file writing. – shivam gupta May 20 '20 at 06:33
  • Also, in my case the snmp walk is giving a Timout: No response.Can this be a reason too?? – shivam gupta May 20 '20 at 06:35
  • @shivamgupta If you're getting a timeout, it means there's nothing on the server to walk and that will "hang" the application yes, obviously. But that goes against your example data, you never mentioned the timeout. The timeout text goes to `stderr` IIRC. – Torxed May 20 '20 at 06:58