1

enter image description hereI am trying scrape data from a AJAX web interface site. Link to that website is as follows:

https://dps.psx.com.pk/

To do so I used the following code:

import requests

request_url = 'https://dps.psx.com.pk/webpages/history_bySymbol.php?op=4'
data = {'sday': '01', 'mth':'10', 'yr':'2012','symcode':'ENGRO'}  
headers = {'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.79 Chrome/61.0.3163.79 Safari/537.36'}
r = requests.post(request_url, data=data, headers=headers)
print(r.text)

Now the problem is when I print the text I am unable to get data which is supposed to be generated by entering 4 parameters.

What might be wrong with the code?

Edit: Issue is that when I open the above webpage, on top there is a drop down menu for market data, from there I select historical data. Link in the chrome bar remains the same since it is an AJAX web interface. My goal is to scrape historical data for that I am using above mentioned code. Using chrome's developer tools I looked at what ajax key was added when I requested for historical data. From there I found out it took four parameters as mentioned in the above code (XHR subsection to Headers). I expect that when I run the code it should provide me with historical data somewhere in r.text, but it doesn't. I need the data as shown in the image. I expected that above code to give these values but it didn't.

Furqan Hashim
  • 1,304
  • 2
  • 15
  • 35

2 Answers2

1
payload = {'sday':'03','mth':'10','yr':'2017'}

r = requests.post(url='https://dps.psx.com.pk/webpages/history_byDate1.php', data=payload, headers ={'X-Requested-With': 'XMLHttpRequest'})

s = BeautifulSoup(r.text, 'lxml')

This worked perfectly and gave results which I wanted.

Furqan Hashim
  • 1,304
  • 2
  • 15
  • 35
0

You must send files instead of data. Take a look:

response = requests.post('http://httpbin.org/post', files=dict(foo='bar'))

By using the files this will force the requests lib to use formdata instead

Also, below is the correct url. I think.

https://dps.psx.com.pk/webpages/history_bySymbol_data.php?op=4
Luke101
  • 63,072
  • 85
  • 231
  • 359
  • The script runs properly but when it prints in the last line. I can see an error Your requrest has been rejected by the following cause:

    Day is required
    – Furqan Hashim Oct 01 '17 at 13:47
  • Then I changed 'sday' to Day but the same error persist. – Furqan Hashim Oct 01 '17 at 13:49
  • changes I made are as follow: data = {'Day': '01', 'Month':'10', 'Year':'2012','Symbol':'Engro CorpXD'} – Furqan Hashim Oct 01 '17 at 13:50
  • I see in postman they only accept multipart/formdata. I got it to work using postman in the form-data section. Dont use a simple ajax and json. – Luke101 Oct 01 '17 at 14:16
  • Since I am new to python I can't get what postman is and where do I need to make changes. if you can kindly guide me where do I need to make change? Or if you can suggest a tutorial or guide that would help me with it. – Furqan Hashim Oct 01 '17 at 14:27
  • I am not familiar with python either. If you like you can ask another question on stackoverflow on how to do this. – Luke101 Oct 01 '17 at 15:09
  • Can you elaborate your above comment on mutlipart/formdata? and what exactly the problem is? – Furqan Hashim Oct 01 '17 at 15:11
  • From here https://stackoverflow.com/questions/12385179/how-to-send-a-multipart-form-data-with-requests-in-python. I believe I've passed a multipart encoded data since my data is in form of dict. – Furqan Hashim Oct 01 '17 at 15:43
  • I editied my answer. But if you use files variable instead of data it will send multipart.formdata instead. This is not tested so you may need to play with it a little. – Luke101 Oct 02 '17 at 01:20
  • Perfect. But I did it with another way I posted request through requests module and then I used beautifulsoup4 to parse which resolved my issue. – Furqan Hashim Oct 02 '17 at 17:34
  • That is great. I am interested in your solution. Please share in your question or post an answer to this question. – Luke101 Oct 03 '17 at 02:38