2

Is it possible to capture output from wget and other command line programs that use curses? Here is what I have right now:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=0)
for line in p.stdout:
    print "a"

This works fine for programs that have simple output, but not for wget and other programs that use curses.

Isaiah
  • 4,201
  • 4
  • 27
  • 40
  • 2
    I don't think wget uses curses. It just prints its messages to stderr rather than stdout. – Jukka Matilainen Aug 17 '09 at 22:57
  • For the case of wget it's probably stderr that you need to intercept (see http://blog.doughellmann.com/2007/07/pymotw-subprocess.html for some examples) For the intercepting the curses programs - I suspect that would become tedious very quickly when you need to emulate the terminal... – Andrew Y Aug 17 '09 at 22:59
  • My bad, I thought that to update an existing line you had to use curses. – Isaiah Aug 17 '09 at 23:18
  • A common means of printing progress bars and spinners in terminal applications is to use the backspace/rubout character. It's best to check that your file descriptor is connected to a terminal (avoid cluttering regular files with ASCII control characters). However, this doesn't require curses. In fact the Python termios module has just what you need for this sort of thing. (Usually, even without the termios module you can sys.stdout.write(chr(0x08)) to backspace over a character on the current line). – Jim Dennis May 18 '10 at 14:39

1 Answers1

7

I don't believe that wget is using curses.

Normally when I want to use wget in a script I'd use the -O - option to force its output to stdout. I suspect you're trying to capture the text that you normally see on your console when you're running it, which would be stderr.

From the command line, outside of Python, just run a command like:

wget -O - http://www.somesite.org/ > /tmp/wget.out 2> /tmp/wget.err

Then look at the two output files. If you see any output from wget on your console/terminal then you are running some different flavor of the command than I've seen.

If, as I suspect, you're actually interested in the stderr messages then you have two choices.

  • Change your command to add 2>&1 and add shell=True to your Popen() arguments
  • Alternatively (and preferably) add stderr=subprocess.PIPE to your Popen() arguments

The former is handy if you weren't using stdout anyway (assuming your using wget to fetch the data and write it into files). In the latter case you read from the stderr file option to get your data.

BTW: if you really did need to capture curses data ... you could try to use the standard pty module but I wouldn't recommend that. You'd be far better off fetching the pexpect module from:

And don't be scared off by the age or version numbering, it works on Python 2.5 and 2.6 as well as 2.4 and 2.3.

Jim Dennis
  • 17,054
  • 13
  • 68
  • 116
  • 1
    Definitely with pexpect, as I recommended in other recent answers -- e.g. see http://stackoverflow.com/questions/1283061 – Alex Martelli Aug 18 '09 at 01:44
  • 1
    Why would I add shell=True? It is not handy at all - and needlessy invokes a new shell process. – nosklo Aug 18 '09 at 03:27
  • I was pointing out that for redirections of the 2>&1 sort you'd either have to do shell=True or you'd have to do your own os.fork(), os.dup2() and os.execve() functions. – Jim Dennis Aug 19 '09 at 12:21
  • You should be getting wget's output before it's "done" (that is before the process has completed). However, it's likely that you're seeing some buffering effects. Try setting the bufsize=0 on the subprocess list. – Jim Dennis Aug 19 '09 at 12:25
  • 1
    @Jim Dennis: you're wrong. subprocess.Popen can do `2>&1` just fine, you don't need to fork anything and it works in all platforms. Just use `subproces.Popen(cmd, stderr=subprocess.STDOUT)` – nosklo Aug 20 '09 at 04:07
  • @nosklo I was referring to the literal syntax 2>&1 as part of the cmd string. Of course there are other ways to get this effect by manipulating the file descriptors (as I said in my original response). – Jim Dennis Nov 28 '16 at 07:08