3

I'm trying to track the tickets trend

currently, I'm using selenium to simulate submitting forms.

as you know, the selenium is slow and consume much more memory.

However, when you submit the form, it will redirect you to a new url http://makeabooking.flyscoot.com/Flight/Select

Therefore, I don't have the idea how could I do this without the selenium.

Because I couldn't change the form of query like this http://makeabooking.flyscoot.com/Flight/from={TPE}&to={NYK}&date={2015-10-12} to fetch the result.

Any idea to do this with Ruby or Python with SSL proxy and HTTP proxy support ?

sample website: http://www.flyscoot.com/index.php/en/

user3675188
  • 7,271
  • 11
  • 40
  • 76
  • Your cart is before your horse. We help you debug code you've written, but you haven't given us any. Please read "[ask]" and "[mcve]". – the Tin Man Dec 17 '15 at 19:58

2 Answers2

2

You can get the curl requests from chrome easily and use it by: 

F12 > Network > request > Right Click > Copy As cURL

curl 'http://makeabooking.flyscoot.com/Flight/Select' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,tr;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8' -H 'Referer: http://www.flyscoot.com/index.php/en/' -H 'Cookie: optimizelyEndUserId=oeu1444666692081r0.12463579000905156; __utmt=1; granify.lasts@1345=1444666699786; ASP.NET_SessionId=lql5yzv1l3yatkh1lcumg2e5; dotrez=1209262602.20480.0000; optimizelySegments=%7B%222335550040%22%3A%22gc%22%2C%222344180004%22%3A%22referral%22%2C%222354350067%22%3A%22false%22%2C%222355380121%22%3A%22none%22%7D; optimizelyBuckets=%7B%223025070068%22%3A%223020800213%22%7D; __utma=185425846.733949751.1444666694.1444666694.1444666694.1; __utmb=185425846.2.10.1444666694; __utmc=185425846; __utmz=185425846.1444666694.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/33084039/how-could-i-query-the-result-without-selenium-on-python-or-ruby; granify.uuid=68b0d8e8-d068-40d8-9068-3098e870b858; granify.session@1345=1444666699786; granify.flags@1345=8; _gr_ep_sent=1; _gr_er_sent=1; granify.session_init@1345=2; optimizelyPendingLogEvents=%5B%5D' -H 'Connection: keep-alive' -H 'X-FirePHP-Version: 0.0.6' -H 'Cache-Control: max-age=0' --compressed

If you can set the headers and cookies info correctly you can use Python requests. If you want to convert it to the Python requests, you can use the this link. By this way you can simulate the browser. See the pyton requests:

cookies = {
    'optimizelyEndUserId': 'oeu1444666692081r0.12463579000905156',
    '__utmt': '1',
    'granify.lasts@1345': '1444666699786',
    'ASP.NET_SessionId': 'lql5yzv1l3yatkh1lcumg2e5',
    'dotrez': '1209262602.20480.0000',
    'optimizelySegments': '%7B%222335550040%22%3A%22gc%22%2C%222344180004%22%3A%22referral%22%2C%222354350067%22%3A%22false%22%2C%222355380121%22%3A%22none%22%7D',
    'optimizelyBuckets': '%7B%223025070068%22%3A%223020800213%22%7D',
    '__utma': '185425846.733949751.1444666694.1444666694.1444666694.1',
    '__utmb': '185425846.2.10.1444666694',
    '__utmc': '185425846',
    '__utmz': '185425846.1444666694.1.1.utmcsr=stackoverflow.com|utmccn=(referral)|utmcmd=referral|utmcct=/questions/33084039/how-could-i-query-the-result-without-selenium-on-python-or-ruby',
    'granify.uuid': '68b0d8e8-d068-40d8-9068-3098e870b858',
    'granify.session@1345': '1444666699786',
    'granify.flags@1345': '8',
    '_gr_ep_sent': '1',
    '_gr_er_sent': '1',
    'granify.session_init@1345': '2',
    'optimizelyPendingLogEvents': '%5B%5D',
}

headers = {
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'en-US,en;q=0.8,tr;q=0.6',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Referer': 'http://www.flyscoot.com/index.php/en/',
    'Connection': 'keep-alive',
    'X-FirePHP-Version': '0.0.6',
    'Cache-Control': 'max-age=0',
}

requests.get('http://makeabooking.flyscoot.com/Flight/Select', headers=headers, cookies=cookies)

If you save the result, you can see that result is as done via browser (open stack.html):

r = requests.get('http://makeabooking.flyscoot.com/Flight/Select', headers=headers, cookies=cookies
f = open("stack1.html", "w")
f.write(r.content)
Mesut GUNES
  • 7,089
  • 2
  • 32
  • 49
  • Hi it seems you must have an `random session value` in the header for the query, So I must get the `random session value` from the FORM page, and then build the query with header and cookie. is it right ? – user3675188 Oct 13 '15 at 00:23
  • Depends on the page, you may righ. Because some of the cookie may be invalidated. – Mesut GUNES Oct 13 '15 at 04:12
0

I think this answer https://stackoverflow.com/a/1196151/1033953 is what you're looking for.

You'll need to inspect the parameters on that form to make sure you're posting the right values, but then you just need to use the Ruby net/http to send the HTTP Post.

I'm sure Python has something similar. Or you could use curl to post as shows in this answer https://superuser.com/a/149335

Community
  • 1
  • 1
Greg
  • 430
  • 1
  • 5
  • 17
  • Hi i will try it later, but wouldn't the redirect action to another domain will break the `net/http` ??? No mater what I query, i will be redirected to another domain `http://makeabooking.flyscoot.com/Flight/Select` – user3675188 Oct 12 '15 at 15:18