Python Requests - Return Response to PHP

Question

I am trying to use Python Requests via PHP for scraping a page
index.php:

<?php
$url = $_GET['url'];
$output = shell_exec("python /home/myusername/public_html/py/fetch.py $url");
echo $output;

and here is my fetch.py:

#! /usr/bin/python

import requests
import sys

url = sys.argv[1]

headers = {'cache-control': 'no-cache'}

try:
    r = requests.request("GET", url, headers=headers)
    #print r.status_code
    #sys.exit(1)
except requests.exceptions.RequestException as e:
    print e
    sys.exit(1)

if not r.text:
    print("Response Empty")
else:
    print(r.text)

I tried checking status code, it's 200. I tried checking if response is empty, it's not. But r.text is not printing at all. What i am doing wrong here?

Is there a reason why you use this two-step process? PHP can do this easily by itself (it will be faster too). — Bart Friederichs, Mar 18 '19 at 07:54
@BartFriederichs the page i am trying to scrape is not working with PHP CURL, don't know why. I tested with Requests, it's working fine in `IDLE` — Sukhchain Singh, Mar 18 '19 at 07:56
Your Python code works perfectly in Python 2.7.15 on my machine. — Bart Friederichs, Mar 18 '19 at 08:01
See here: https://stackoverflow.com/questions/4214031/php-and-shell-exec — zvi, Mar 18 '19 at 08:02
@BartFriederichs yes it works, but when i am executing this via PHP, it is returning empty response.. :( — Sukhchain Singh, Mar 18 '19 at 08:02
I ran this through PHP as well, and it gives a `UnicodeEncodeError`. — Bart Friederichs, Mar 18 '19 at 08:03
@BartFriederichs Thanks for info it's useful, what can be the possible solution for this? — Sukhchain Singh, Mar 18 '19 at 08:06

zvi · Answer 1 · 2019-03-18T08:07:58.640

1

Trying this way:

shell_exec("python /home/myusername/public_html/py/fetch.py $url 2>&1");

you can see the errors in the python script you have.

edited Mar 18 '19 at 08:07

answered Mar 18 '19 at 07:58

zvi

3,677
2
30
48

`Traceback (most recent call last): File "/home/myusername/public_html/py/ig.py", line 21, in print(r.text) UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 15318: ordinal not in range(128)` – Sukhchain Singh Mar 18 '19 at 08:00

score 1 · Accepted Answer · answered Mar 18 '19 at 08:07

According to the official documentation there are some catches included in shell_exec

You may have to update your code a bit. There are chances that you may have error in your code, which are basically redirected to stderr and hence not captured in the stdout. Further read this

So back to your question. Edit your code

<?php
$url = $_GET['url'];
$output = shell_exec("python /home/myusername/public_html/py/fetch.py $url 2>&1");
echo $output;
?>

Update Python code to:

#! /usr/bin/python

import requests
import sys

url = sys.argv[1]

headers = {'cache-control': 'no-cache'}

try:
    r = requests.request("GET", url, headers=headers)
    #print r.status_code
    #sys.exit(1)
except requests.exceptions.RequestException as e:
    print e
    sys.exit(1)

if not r.text:
    print("Response Empty")
else:
    print(r.text.encode('utf-8')) # changes

@BartFriederichs i guess may be some contents in the page may be encoded according to some region. We can never guess how the website behaves. — Ja8zyjits, Mar 18 '19 at 08:39

Python Requests - Return Response to PHP

2 Answers2