I am trying to run an automation script (python-selenium script) that scrapes data from websites after taking some input from the client side. I am trying to send the credentials using a form and process it then using shell_exec or exec or symfony/process library. So far nothing works for me.
This is my form -
@extends('layouts.app')
@section('content')
<div class="container">
@include('includes.messages')
<h1>Create Scraper</h1>
{!! Form::open(['action' => 'ScraperController@store', 'method' => 'POST', 'novalidate' => 'novalidate']) !!}
<div class="form-group">
{{ Form::label('url', 'Search URL') }}
{{ Form::text('url', '', ['class' => 'form-control', 'placeholder' => 'Search URL']) }}
</div>
<div class="form-group">
{{ Form::label('email', 'Email') }}
{{ Form::email('email', '', ['class' => 'form-control', 'placeholder' => 'Email']) }}
</div>
<div class="form-group">
{{ Form::label('password', 'Password') }}
{{ Form::password('password', ['class' => 'form-control', 'placeholder' => 'Password']) }}
</div>
{{ Form::submit('Submit', ['class' => 'btn btn-primary']) }}
{!! Form::close() !!}
</div>
@endsection
This is my controller -
I have tried it with process library it gives nothing no error, no response executes the below code straight away but scraper run unsuccessful
use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;
public function store(Request $request) {
$this->validate($request, [
'url' => 'required',
'email' => 'required|email:rfc,dns',
'password' => 'required'
]);
$process = new Process(['C:\xampp\htdocs\scraper\public\lkscraper\venv\Scripts\python', 'C:\xampp\htdocs\scraper\public\lkscraper\lk.py']);
$process->run();
if (!$process->isSuccessful()) {
throw new ProcessFailedException($process);
}
echo $process->getOutput();
}
I have also tried the following - I have made all the dependency global in my machine an run it. But again not successful in laravel. But the same shell_exec run fine and I get the desired output when run as single file outside the laravel project.
public function store(Request $request) {
$this->validate($request, [
'url' => 'required',
'email' => 'required|email:rfc,dns',
'password' => 'required'
]);
$output = shell_exec("python C:\xampp\htdocs\scraper\public\lkscraper\lk.py");
var_dump($output);
}
When I run the following keeping inside a seperate php file it works fine. But inside laravel it is not working
$output = shell_exec("python C:\xampp\htdocs\scraper\public\lkscraper\lk.py");
var_dump($output);
Please help me solve this. What am I doing wrong! Thank you for your time
The Error I am getting:
string(1418) "Traceback (most recent call last):
File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\common\service.py", line 72, in start
self.process = subprocess.Popen(cmd, env=self.env,
File "C:\Users\raman\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\raman\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1307, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\xampp\htdocs\scraper/lkscraper/lk.py", line 15, in
browser = webdriver.Chrome('driver/chromedriver.exe')
File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "C:\xampp\htdocs\scraper\lkscraper\venv\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: 'chromedriver.exe' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home