How to pass parameter to Url with Python urlopen

Question

I'm currently new to python programming. My problem is that my python program doesn't seem to pass/encode the parameter properly to the ASP file that I've created. This is my sample code:

import urllib.request

url = 'http://www.sample.com/myASP.asp'
full_url = url + "?data='" + str(sentData).replace("'", '"').replace(" ", "%20").replace('"', "%22") + "'"
print (full_url)
response = urllib.request.urlopen(full_url)
print(response)

the output would give me something like:

http://www.sample.com/myASP.asp?data='{%22mykey%22:%20[{%22idno%22:%20%22id123%22,%20%22name%22:%20%22ej%22}]}'

The asp file is suppose to insert the acquired querystring to a database.. But whenever I check my database, no record is saved. Though if I do copy and paste the printed output on my browser url, the record is saved. Any input on this? TIA

Update: Is it possible the python calls my ASP File A but it doesn't call my ASP File B? ASP File A is called by python while ASP File B is called by ASP File A. Because whenever I run the url on a browser, the saving goes well. But in python, no saving of database occurs even though the data passed from python is read by ASP File A..

A wild guess: If the output url works in a browser, please verify if the browser actually does more work than your script: Maybe the content returned by the asp server script triggers more requests in your browser which are required to trigger the database writing on the server side. This could be checked by looking at the output of the `print(response)` and checking it for more requests. — cfi, Nov 08 '12 at 13:28
Yes it does. The asp file that my python file is calling calls another asp file. Its something like this: my Python file calls Asp File A and Asp File A calls ASP File B when it is triggered. — Eduard, Nov 08 '12 at 14:22
Sry, my bad communication: The output of the response you get not with `print(response)` but by iterating over it. The docs of `urllib` say that the object behaves like a context manager, so you can do sth like `with urllib.request.urlopen(....) as web_page: for line in web_page: print(line)` to look at the content returned to you. Insert line breaks and proper indents.. — cfi, Nov 08 '12 at 16:03

score 3 · Answer 1 · answered Nov 08 '12 at 14:46

3

Use firebug with Firefox and watch the network traffic when the page is loaded. If it is actually an HTTP POST, which I suspect it is, check the post parameters on that post and do something like this:

from BeautifulSoup import BeautifulSoup
import urllib

post_params = {
              'param1' : 'val1',
              'param2' : 'val2',
              'param3' : 'val3'
              }
post_args = urllib.urlencode(post_params)

url = 'http://www.sample.com/myASP.asp'
fp = urllib.urlopen(url, post_args)
soup = BeautifulSoup(fp)

If its actually HTTP POST, this will work.

answered Nov 08 '12 at 14:46

That1Guy

7,075
4
47
59

will this affect the asp file? Because I am currenty using request.querystring.. should I change it to Request.form? – Eduard Nov 08 '12 at 14:58
I'm not sure about that, to be honest. It couldn't hurt. Give it a try and let me know how it works out! =) – That1Guy Nov 08 '12 at 15:09
btw, could you please explain the last part `soup = BeautifulSoup(fp)` ? I can't seem to understand what this does.. – Eduard Nov 08 '12 at 15:18
Oh, sorry. BeautifulSoup is an html parsing library for python. It makes parsing web pages a lot easier. The documentation can be found at 'http://www.crummy.com/software/BeautifulSoup/'. Also, if this helped, I'd like it if you accepted this answer as correct. No pressure, but it helps out my rep =) – That1Guy Nov 08 '12 at 15:52
Yes I'll do that once I've verified the code. Thanks a ton bro – Eduard Nov 08 '12 at 15:56
No problem. I'm here to help. If there is anything else I can do, just leave a comment and I'll do my best. – That1Guy Nov 08 '12 at 15:58
It is a `GET`, and even though I changed it to `POST`, still no data is processed. I even tried changing the charset upon retrieving the values as utf-8 on my asp file – Eduard Nov 09 '12 at 04:20
Hmmm...If this was a page I was parsing, at this point I would use selenium. Its hard to figure out what is actually going on without viewing the actual page. If I knew what was going on, I'd be more likely to figure it out. I'd recommend using Selenium for this, though. See my answer on this question: http://stackoverflow.com/questions/13039530/unable-to-call-firefox-from-selenium-in-python-on-aws-machine/13055412#13055412 for some insight as to how to proceed. Also, the Selenium documentation can be found here: http://seleniumhq.org/ This WILL do the job if you use it correctly! =) – That1Guy Nov 09 '12 at 14:42

score 1 · Accepted Answer · answered Aug 23 '16 at 08:23

In case anybody stumbles upon this, this is what I've come up with:

py file:

url = "my.url.com"
data = {'sample': 'data'}

encodeddata = urllib.parse.urlencode(data).encode('UTF-8')
req = urllib.request.Request(url, encodeddata)
response = urllib.request.urlopen(req)

and in my asp file, I used json2.js:

jsondata = request.form("data")
jsondata = replace(jsondata,"'","""")
SET jsondata = JSON.parse(jsontimecard)

Note: use requests instead. ;)

score -1 · Answer 3 · edited Jun 20 '20 at 09:12

-1

First off, I don't know Python.

But from this : doc on urllib.request

the HTTP request will be a POST instead of a GET when the data parameter is provided

Let me make a really wild guess, you are accessing the form values as Request.Querystring(..) in the asp page, so your post wont pass any values. But when you paste the url in the address bar, it is a GET and it works.

just guessing, you could show the .asp page for further check.

edited Jun 20 '20 at 09:12

Community

1
1

answered Nov 08 '12 at 14:36

Flakes

2,422
8
28
32

Yes you are correct. The asp file has a Request.Querystring(..) that gets the sample data parameter string which is '{%22mykey%22:%20[{%22idno%22:%20%22id123%22,%20%22name%22:%20%22ej%22}]}'.. Btw, I believe that I did not provide a data parameter on my urlopen.. or am I wrong? Because from what I've read, the structure when passing data is something like this: response = urllib.request.urlopen(full_url, data) – Eduard Nov 08 '12 at 14:58
from your post: `full_url = url + "?data='" + str(sentData)...` – Flakes Nov 08 '12 at 15:00
yes but the syntax used on urllib.request.urlopen() contains only 1 parameter, which is the url. `response = urllib.request.urlopen(full_url)` – Eduard Nov 08 '12 at 15:16
but your `full_url` contains the data . Just change the `Request.Querystring(..)` to `Request(..)` and see. – Flakes Nov 09 '12 at 06:24
`Request()`? Doesn't ASP use `Request.Querystring` for `GET` and `Request.Form` for `POST` ? Sorry I'm not really familiar with ASP.. – Eduard Nov 09 '12 at 06:40
just Request(..) will also work. It first looks in the querystring, then the form , then the cookies ... .It's just a shortcut. – Flakes Nov 09 '12 at 06:47
[from this link :](http://oreilly.com/catalog/aspnut/chapter/ch06.html) `strName = Request("name") returns the value of the "name" key regardless of the collection in which it's located, because IIS searches all collections. When you specify a value in this manner, ASP looks through each Request object collection in the following order:...` – Flakes Nov 09 '12 at 06:52

How to pass parameter to Url with Python urlopen

3 Answers3

Linked