0

I have an executable (named "prova"), that has two inputs and run via single bash command line:

./prova ../MW1/atmf370r10a Ctes370r10a

Then the first input is a path to a file I need to read and the second is one to be created. This executable has to be run over a lot of files, so a loop is needed.

My python script reads a list of files named "filename" and runs a loop:

with open(filename) as f:
  for line in f:

       AtmFileName=line.split(None, 1) 

       if (is_number(AtmFileName[0])==False):

           MassFileName=AtmFileName[1]
           AtmFileName= AtmFileName[0] 
           bashCommand="./prova ../MW1/"+AtmFileName+" C"+MassFileName
           print "bash command is: "+ bashCommand      
           os.system(bashCommand)
       if 'str' in line:
           break

It runs. The files are created. But, they are appended by ? at the end of the name. I can see the marks only if I type ls. Those files can't be opened.

If I run just one line by manually copying one of printed lines ("bash command is: " ) it works and the files are not appended by ? and they can be opened. What can be the problem?

Paul Dawson
  • 1,332
  • 14
  • 27
Michael L.
  • 443
  • 4
  • 14
  • Using `os.system()` this way is inherently insecure; Python's `subprocess` module, with the default `shell=False`, is much safer. – Charles Duffy Jul 16 '19 at 15:37
  • As for the `?`s, that typically means your input file has DOS newlines. Run `printf '%q\n' *`, and you'll see the names printed as `$'whatever\r'`; a DOS newline is `\r\n`, and a UNIX newline is `\n`, so when you read a file with DOS newlines in UNIX, it looks like each file has a `\r` on the end. – Charles Duffy Jul 16 '19 at 15:38
  • ...nothing to do with `os.system()`, or with Python, or even with bash; you can get this same problem in any language, if you're parsing a DOS text file as if it were a UNIX one. – Charles Duffy Jul 16 '19 at 15:39
  • ...that said, consider `subprocess.call(['./prova', '../MW1/' + AtmFileName.rstrip('\r'), 'C' + MassFileName.rstrip('\r')])` -- though even that isn't really safe unless you sanitize your filenames (making sure they can't contain `/`s, for example), it's much *less* unsafe than the original, which would treat a filename containing `$(rm -rf ~)` as an instruction to delete all your files. – Charles Duffy Jul 16 '19 at 15:42
  • 1
    (Granted, if you're splitting on all whitespace recognized by your shell that particular case can't happen without some creativity, but `$(/tmp/foo)` lets someone who can create files in `/tmp` -- which even untrusted users can do -- run code in your account, so it's a privilege escalation / lateral-movement attack either way). – Charles Duffy Jul 16 '19 at 15:44
  • Thanks to Charles Duffy for the answer. Can't say I understood everything, but if I got it correctly, you say that the file I read was created in DOS system and Linux got some trouble with reading it because of some new-line character. So I created a new list file and just copied the text form previous on there. Worked perfectly! Thanks. As for security, I not so sure what does it mean: safety against what? I am the only user who will run this code. – Michael L. Jul 16 '19 at 16:04
  • Safety against someone malicious creating the filenames in the data files you run the code against. Even if you're the only one running it, did you create and audit all the data yourself? – Charles Duffy Jul 16 '19 at 16:06
  • 1
    And it's also worth being in good habits when it doesn't matter, so that you don't get bit when it unexpectedly *does* matter; the worst data loss incident I've been present for was caused by a bug in a C library used by a Python module in 1st-party code dumping random memory into filenames that "couldn't ever" contain anything but hex digits, which were later processed by a sloppily-written shell script; sure, that's a once-in-a-decade event, but it was expensive enough that having the company's ops staff spending the decade prior writing careful, safe scripts would have been worth the cost. – Charles Duffy Jul 16 '19 at 16:07

0 Answers0