1

I'm trying to execute curl command in python script and couldn't pass a password with symbols.

import os;
os.system("curl -H 'Content-Type: text/xml;charset=UTF-8' -u 'appuser:appuser!@3pass' -i -v 'http://app.com/webservice/getUserData' -o userdata.xml")

And I get Access Denied message in return, the username and password are correct. I guess it is because of the special characters in the password. I have tried escape the characters like appuser\!\@3pass but didn't help.

Can anyone guide through this?

Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
beginner
  • 183
  • 1
  • 5
  • 13

1 Answers1

4

the command you're using cannot work because single quotes have no special meaning in Windows, they're passed literally to the curl program. So if you copied the command line from Linux (where it would work), that won't work here (spurious quotes are passed to curl for instance for the login/password field which recieves 'appuser and appuser!@3pass', also Content-Type: text/xml;charset=UTF-8 isn't protected at all and is understood as 2 separate arguments)

Simple test from the console:

K:\progs\cli>curl.exe -V
curl 7.50.3 (x86_64-pc-win32) libcurl/7.50.3 OpenSSL/1.0.2h nghttp2/1.14.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp smb smbs smtp smtps
telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL HTTP2

(also works if I use "-V" with double quotes), but if I use simple quoting on the version arg I get:

K:\progs\cli>curl.exe '-V'
curl: (6) Could not resolve host: '-V'

As o11c commented, there's a python module to handle curl, you'd be better off with it.

For other cases when you can't do otherwise, using os.system is deprecated. Using subprocess.check_call for instance is better (python 3.5 has a unified run function):

  • return code checked, exception raised if error
  • ability to pass & quote arguments without doing it manually.

Let's fix your example:

subprocess.check_call(["curl","-H","Content-Type: text/xml;charset=UTF-8","-u",'appuser:appuser!@3pass',"-i","-v",'http://app.com/webservice/getUserData',"-o","userdata.xml"])

note that I purposedly mixed single & double quotes. Python doesn't care. And if there's a space in the argument, check_call mechanism manages to handle argument protection/quoting automatically.

when calling this script I get userdata.xml filled in like:

HTTP/1.1 301 Moved Permanently
Date: Fri, 15 Sep 2017 20:49:50 GMT
Server: Apache
Location: http://www.app.com/webservice/getUserData
Content-Type: text/html; charset=iso-8859-1
Transfer-Encoding: chunked

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="http://www.app.com/webservice/getUserData">here</a>.</p>
</body></html>
Jean-François Fabre
  • 137,073
  • 23
  • 153
  • 219
  • On Windows an args list is converted to a command line string via `subprocess.list2cmdline`, which assumes the application will follow the parsing rules that are common to [VC++](https://msdn.microsoft.com/en-us/library/17w5ykft) and [`CommandLineToArgvW`](https://msdn.microsoft.com/en-us/library/bb776391). curl.exe probably follows these rules. Most applications do. IIRC, some Cygwin executables use different rules. – Eryk Sun Sep 15 '17 at 20:11
  • @eryksun I'm not sure of that. Actually I read that list2cmdline only emulated the way the command is sent to the system, but that you cannot rely on that. you're right about execv though. I'll edit it out – Jean-François Fabre Sep 15 '17 at 20:13
  • @eryksun I'm lucky enough to have a version of curl on my PC. I tested simple quotes and they're passed literally. curl isn't able to strip them off. – Jean-François Fabre Sep 15 '17 at 20:20
  • here: https://stackoverflow.com/questions/14836947/output-the-command-line-called-by-subprocess one answer implies that list2cmdline "does its best". – Jean-François Fabre Sep 15 '17 at 20:26
  • @Jean-FrançoisFabre Thanks so much for the response. Now, I'm getting Access Forbidden error but when I access the url on a web browser I can fetch the data. – beginner Sep 15 '17 at 20:35
  • have you tried to type the exact command line of your question in a cmd shell? does _that_ work? – Jean-François Fabre Sep 15 '17 at 20:46
  • `list2cmdline` uses simple rules that work with a VC++ application. For the example in the [linked question](https://stackoverflow.com/questions/14836947), `cmd "--arg=foo bar" baz` and `cmd --arg="foo bar" baz` are equivalent by these rules. If an app doesn't handle these command lines equivalently, then it's not compatible with `list2cmdline`. – Eryk Sun Sep 15 '17 at 20:51
  • @eryksun understood. The issue here is that simple quotes are just ordinary characters. They don't even protect against spaces. – Jean-François Fabre Sep 15 '17 at 20:52
  • @Jean-FrançoisFabre Yes, the same command works fine on git bash – beginner Sep 15 '17 at 20:55
  • @Jean-FrançoisFabre it was missing the cookie it is working :) – beginner Sep 15 '17 at 21:00
  • 1
    Some applications such as schtasks.exe actually have special parsing for *both* single and double quotes. My first comment was only meant to clarify what `Popen` is doing for an args list. Whether or not what it does works for curl.exe is a matter of experiment, which you did, or running it under a debugger to check whether it uses the `argv` array from Microsoft's CRT, `CommandLineToArgvW`, or some custom command-line parsing. The lack of consistency is unfortunate, but that's Windows... – Eryk Sun Sep 15 '17 at 21:05