2

I want to use the "raster2pgsql" utility in my Python code. When I use it in a Linux terminal, it works fine. This is the command:

$ raster2pgsql -a "/mnt/c/Users/Jan/path/to/raster/dem.tiff" test_schema.raster2 | psql -h localhost -d pisl -U pisl

Then I use subprocess.run (I have also tried subprocess.call) to use this same tool in my Python code. This is my code:

from subprocess import run
command = ["raster2pgsql", "-a", '"' + file_name + '"', self.schema_name +  "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command)

I get this error:

ERROR: Unable to read raster file: "/mnt/c/Users/Jan/path/to/raster/dem.tiff"

Printing command gives this which I think is correct (equivalent to what worked in the terminal):

['raster2pgsql', '-a', '"/mnt/c/Users/Jan/path/to/raster/dem.tiff"', 'test_schema.raster2', '|', 'psql', '-h', 'localhost', '-p', '5432', '-d', 'pisl']

I have double checked that the path to the raster file is correct, tried single quotes, double quotes but nothing helps. I have looked at a number of similar question (here, here or here ) but did not find anything helpful.

I use Python 3.5 and Linux Bash Shell in Windows 10.

Question: What is wrong with the way I use subprocess?

Jan Pisl
  • 1,043
  • 2
  • 14
  • 29
  • 2
    why are you quoting the filename at all? – Jean-François Fabre Jun 27 '18 at 18:43
  • because it doesn't recognize the filename when I don't. When I remove the quotes it considers "test_schema.raster2" to be the filename. – Jan Pisl Jun 27 '18 at 18:45
  • I've provided an alternate method with 2 commands piped together. Please test that. report error accurately if any. I doubt that the quotes are the issue, since the shell removes them on the working command you performed. – Jean-François Fabre Jun 27 '18 at 18:48

1 Answers1

4

2 issues here:

  • no need to extra-quote the filename. It's passed to the system literally and since there's no file called "/tmp/something" the command fails.
  • second, to be able to pass a pipe, you need shell=True

so quickfix:

command = ["raster2pgsql", "-a", file_name, self.schema_name +  "." + identifier, "|", "psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command,shell=True)

or using a command string (because shell=True is picky with argument list):

command = "raster2pgsql -a "+ file_name + " " + self.schema_name +  "." + identifier + " | psql -h localhost -p 5432 -d" + self.dbname
run(command,shell=True)

(ugly, isn't it?)

It's much better to run 2 processes without shell=True and pipe them together using python, more portable & secure (not sure how shell=True reacts with an argument list on Linux):

from subprocess import *
command1 = ["raster2pgsql", "-a", file_name, self.schema_name +  "." + identifier]
p = Popen(command1,stdout=PIPE)
command2 = ["psql", "-h", "localhost", "-p", "5432", "-d", self.dbname]
run(command2,stdin=p.stdout)

The first Popen object created writes its output to a pipe (thanks to stdout=PIPE argument). The run function can take an input as a pipe too (thanks to stdin=p.stout). It consumes the output of the first command, creating a native piped chain of commands, without the need of the shell (and the caveats of quoting, spaces, special character interpretation and such...)

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • Thanks the for the explaination. I was trying to ask a py 3 script to run a py 2 script and I use `process = subprocess.run(['py', '-2', os.path.basename(latest_file)])` but for some reason it didn't run. I accidentally came this and by adding `shell=True` in a different line, it works. Any chance you know why it works? Thank you so much! – Donald Li Jan 27 '20 at 21:54
  • you mean that `subprocess.run(['py', '-2', os.path.basename(latest_file)],shell=True)` runs and without shell=True it doesn't? that may well be another question to ask. – Jean-François Fabre Jan 27 '20 at 21:57
  • Oh quite the contrary. In one line it didnt' work. But in the format you proposed as a "dirty": `command = ['py', '-2', os.path.basename(latest_file)] subprocess.run(command, shell=True)`it works. – Donald Li Jan 28 '20 at 19:27
  • @Donald Li In my case if I use a Shell=True I have to use a string and not a list... – Vincent Ricosti Apr 21 '22 at 10:33