1

I am trying to simple file copy operation using dd under Windows. The code is simple. Just open the command using subprocess and read stderr. Then process the stderr to show the progress of the operation. Here is the sample/simple code which I an trying to accomplish:-

import time
import signal
import subprocess
import os

def dd_win():
    windd = "dd.exe"
    in_file = os.path.join("E:", "test-in.iso")
    out_file = os.path.join("E:", "test-out.iso")
    parameter1 = "if=" + in_file
    parameter2 = "of=" + out_file
    parameter3 = "bs=1M"
    parameter4 = "--progress"
    command = [windd, parameter1, parameter2, parameter4]
    print command
    dd_process = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True, bufsize=0)

    while dd_process.poll() is None:
        time.sleep(.1)
        line = dd_process.stderr.readline()
        if not line:
            break
        print ">>>>>", line.strip()
    if dd_process.poll() is not None:
        print "Process finished."

dd_win()

Here is wht I got as an output. I could see the stderr but unable to process the transfer bytes to convert in to mega bytes:-

C:\Users\user\Documents>python test.py
['dd.exe', 'if=E:test-in.iso', 'of=E:test-out.iso', '--progress']
>>>>> rawwrite dd for windows version 0.6beta3.
>>>>> Written by John Newbigin <jn@it.swin.edu.au>
>>>>> This program is covered by terms of the GPL Version 2.
>>>>>
175,882,240  # <--- Problem output line. Unable to assign to other variable for progress calculation
Process finished.

I actually do not know where the numbers are coming from. Is it from stderr or buffer? If it is from stderr then ">>>>>" should have be printed before the numbers. I work on Windows 7 and using windows version of dd.

Any help is appreciated.

sundar_ima
  • 3,604
  • 8
  • 33
  • 52
  • If you remove the print statements from your code, does the `175,882,240` still appear? Can you redirect stdout and stderr from your program to files so that you can see which stream `175,882,240` is coming from? – Tom Dalton Mar 10 '15 at 15:48
  • If I remove `print ">>>>>", line.strip()` statement then I can not see any output on the screen. – sundar_ima Mar 10 '15 at 15:57
  • If you run your dd command in your terminal, do you see "175,882,240" printed, or is it only when you execute it in Python? – skrrgwasme Mar 10 '15 at 16:03
  • I got an error that `close_fds` is not supported on windows platform. – sundar_ima Mar 10 '15 at 16:15
  • Ignore my earlier comment about close_fds - it's set to True by default anyway, tho if Windows doesn't support it it might be related. – Tom Dalton Mar 10 '15 at 16:17
  • It's ?possible? that the line is weird because it contains a ton of control characters. E.g. If dd provides a progressbar or something the output might look like `1\r2\r3\r...\r\n175\n`. I can't work out exactly how that would break readline, but you might be helped by printing out the length of each line (control characters will inflate the length) and/or printing out `",".join(ind(char) for char in line)` or similar. This would at least rule out the possibility of this happening, though I'm clutching at strawas a little here tho! – Tom Dalton Mar 10 '15 at 16:18
  • What is `ind` in the statement you have written `",".join(ind(char) for char in line)` – sundar_ima Mar 10 '15 at 16:33
  • Ah, sorry, I meant `ord(char)`. Ref https://docs.python.org/2/library/functions.html#ord – Tom Dalton Mar 10 '15 at 16:34
  • I got `TypeError: sequence item 0: expected string, int found` error. – sundar_ima Mar 10 '15 at 16:36
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/72688/discussion-between-tom-dalton-and-sundar-ima). – Tom Dalton Mar 10 '15 at 16:37

1 Answers1

1

As discussed in chat, the numbers are coming from stderr. By printing the ascii-indexes of each character in line, we discovered that the final line returned by readline() is \t75,881,728 \r175,882,240 \r\n. It looks like the \r embedded in the middle of this string (which DD outputs) is confusing your code.

Tom Dalton
  • 6,122
  • 24
  • 35
  • OP could enable `universal_newlines=True` to treat `\r` as a newline. `\r` may be used to report progress (e.g., [see how `reporthook()` uses `\r`](http://stackoverflow.com/a/13895723/4279)) -- it may instruct terminal to move the cursor to the beginning of the line. – jfs Mar 11 '15 at 12:40