1

I am trying to run an automation script (python-selenium script) that scrapes data from websites after taking some input from the client side. I am trying to send the credentials using a form and process it then using shell_exec or exec or symfony/process library. So far nothing works for me.

This is my form -

@extends('layouts.app')
@section('content')
    <div class="container">
        @include('includes.messages')
        <h1>Create Scraper</h1>
        {!! Form::open(['action' => 'ScraperController@store', 'method' => 'POST', 'novalidate' => 'novalidate']) !!}
            <div class="form-group">
                {{ Form::label('url', 'Search URL') }}
                {{ Form::text('url', '', ['class' => 'form-control', 'placeholder' => 'Search URL']) }}
            </div>

            <div class="form-group">
                {{ Form::label('email', 'Email') }}
                {{ Form::email('email', '', ['class' => 'form-control', 'placeholder' => 'Email']) }}
            </div>

            <div class="form-group">
                {{ Form::label('password', 'Password') }}
                {{ Form::password('password', ['class' => 'form-control', 'placeholder' => 'Password']) }}
            </div>

            {{ Form::submit('Submit', ['class' => 'btn btn-primary']) }}
        {!! Form::close() !!}
    </div>
@endsection

This is my controller -

I have tried it with process library it gives nothing no error, no response executes the below code straight away but scraper run unsuccessful

use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;

public function store(Request $request) {
    $this->validate($request, [
        'url' => 'required',
        'email' => 'required|email:rfc,dns',
        'password' => 'required'
    ]);

    $process = new Process(['C:\xampp\htdocs\scraper\public\lkscraper\venv\Scripts\python', 'C:\xampp\htdocs\scraper\public\lkscraper\lk.py']);
    $process->run();
    if (!$process->isSuccessful()) {
        throw new ProcessFailedException($process);
    }
    echo $process->getOutput();        
}

I have also tried the following - I have made all the dependency global in my machine an run it. But again not successful in laravel. But the same shell_exec run fine and I get the desired output when run as single file outside the laravel project.

public function store(Request $request) {
    $this->validate($request, [
        'url' => 'required',
        'email' => 'required|email:rfc,dns',
        'password' => 'required'
    ]);

    $output = shell_exec("python C:\xampp\htdocs\scraper\public\lkscraper\lk.py");
    var_dump($output);       
}

When I run the following keeping inside a seperate php file it works fine. But inside laravel it is not working

$output = shell_exec("python C:\xampp\htdocs\scraper\public\lkscraper\lk.py");
var_dump($output);

Please help me solve this. What am I doing wrong! Thank you for your time

The Error I am getting:

string(1418) "Traceback (most recent call last):
  File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\common\service.py", line 72, in start
    self.process = subprocess.Popen(cmd, env=self.env,
  File "C:\Users\raman\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\raman\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\xampp\htdocs\scraper/lkscraper/lk.py", line 15, in 
    browser = webdriver.Chrome('driver/chromedriver.exe')
  File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
    self.service.start()
  File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start
    raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: 'chromedriver.exe' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
The Sorcerer
  • 75
  • 1
  • 7

0 Answers0