1

When I run the below code it returns an empty string

url = 'http://www.allflicks.net/wp-content/themes/responsive/processing/processing_us.php?draw=5&columns[0][data]=box_art&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=false&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=title&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=true&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=year&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=true&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=genre&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=true&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=rating&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=true&columns[4][search][value]=&columns[4][search][regex]=false&columns[5][data]=available&columns[5][name]=&columns[5][searchable]=true&columns[5][orderable]=true&columns[5][search][value]=&columns[5][search][regex]=false&columns[6][data]=director&columns[6][name]=&columns[6][searchable]=true&columns[6][orderable]=true&columns[6][search][value]=&columns[6][search][regex]=false&columns[7][data]=cast&columns[7][name]=&columns[7][searchable]=true&columns[7][orderable]=true&columns[7][search][value]=&columns[7][search][regex]=false&order[0][column]=5&order[0][dir]=desc&start=0&length=25&search[value]=sherlock&search[regex]=false&movies=true&shows=true&documentaries=true&rating=netflix&_=1451768717982'
print requests.get(url).text

but if I put the url in my browser it'll show my the json information. I did notice while debugging the browser must have the plugin tamper data installed to view the json. If a browser doesn't have the plug in a blank web page will appear. So, my theory is it has to do something with out the http request is being handed but I'm not where to go from here.

Any help would be great.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
msaldivar
  • 45
  • 2
  • 8
  • It seems I have no output in browser, are you sure it's the url you want? – pythad Jan 03 '16 at 05:48
  • @pythad if you would open the home allflicks page with browser developer tools and filter the XHR requests in the network tab, you will see the request the OP is asking about, thanks. – alecxe Jan 03 '16 at 05:56

1 Answers1

4

You need to open up a session, visit the main page to get the cookies set and then make the XHR request to "processing_us.php":

url = "http://www.allflicks.net/wp-content/themes/responsive/processing/processing_us.php?draw=5&columns[0][data]=box_art&columns[0][name]=&columns[0][searchable]=true&columns[0][orderable]=false&columns[0][search][value]=&columns[0][search][regex]=false&columns[1][data]=title&columns[1][name]=&columns[1][searchable]=true&columns[1][orderable]=true&columns[1][search][value]=&columns[1][search][regex]=false&columns[2][data]=year&columns[2][name]=&columns[2][searchable]=true&columns[2][orderable]=true&columns[2][search][value]=&columns[2][search][regex]=false&columns[3][data]=genre&columns[3][name]=&columns[3][searchable]=true&columns[3][orderable]=true&columns[3][search][value]=&columns[3][search][regex]=false&columns[4][data]=rating&columns[4][name]=&columns[4][searchable]=true&columns[4][orderable]=true&columns[4][search][value]=&columns[4][search][regex]=false&columns[5][data]=available&columns[5][name]=&columns[5][searchable]=true&columns[5][orderable]=true&columns[5][search][value]=&columns[5][search][regex]=false&columns[6][data]=director&columns[6][name]=&columns[6][searchable]=true&columns[6][orderable]=true&columns[6][search][value]=&columns[6][search][regex]=false&columns[7][data]=cast&columns[7][name]=&columns[7][searchable]=true&columns[7][orderable]=true&columns[7][search][value]=&columns[7][search][regex]=false&order[0][column]=5&order[0][dir]=desc&start=0&length=25&search[value]=sherlock&search[regex]=false&movies=true&shows=true&documentaries=true&rating=netflix&_=1451768717982"
with requests.Session() as session:
    session.headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}
    session.get("http://www.allflicks.net/")

    response = session.get(url, headers={"Accept" : "application/json, text/javascript, */*; q=0.01", 
                                         "X-Requested-With": "XMLHttpRequest", 
                                         "Referer": "http://www.allflicks.net/", 
                                         "Host": "www.allflicks.net"})
    print(response.json())
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195