how do you fix the executable error with chromedriver Windows >>> in a transition to Ubuntu?

Question

My goal is to execute a script i've used previously on windows, transitioned to chromedriver and there's this error:

Blockquote selenium.common.exceptions.WebDriverException: >> Message: 'chromedriver' executable needs to be in the PATH.

the windows exe file is in the PATH and works with windows. I've replaced that chromedriver with an up to date version of Linux chromedriver, have installed chrome web browser to the default directory. it seems to not recognize that the file is there and in the code it's not specified that it's a .exe it's looking for.

Here is the code

import scrapy
from ..items import AuctionscrapingItem
from selenium.webdriver.support.ui import 
WebDriverWait
from selenium.webdriver.support import 
expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import 
TimeoutException
from selenium.common.exceptions import 
NoSuchElementException
from scrapy.crawler import CrawlerProcess
import time
import re
from twisted.internet import reactor, defer
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.settings import Settings
from auctionscraping import settings as my_settings
import mysql.connector
from datetime import datetime

class AuctionInt(scrapy.Spider):
    name = 'auction'
    start_urls = ['https://www.websitefunfirst.com/']


    def __init__(self):
        self.driver = webdriver.Chrome('/opt/lampp/htdocs/scrapy/chromedriver')
        self.driver.maximize_window()
        self.conn = mysql.connector.connect(
            host= 'localhost',
            user= 'root',
            passwd= '',
            database= 'auction'
        )

        self.curr = self.conn.cursor(buffered=True)
        self.curr.execute("""create table IF NOT EXISTS auction_tb(
            id int,
            web_name text,
            website text,
            name text,
            start text,
            end text,
            locations text,
            location_link text,
            remaining_time text,
            current_bid text,
            viewer text,
            premium text,
            image_urls text,
            bid text,
            starting_bid text,
            description text
        )""")
        self.curr.execute("""create table IF NOT EXISTS bid_history(
            id int NOT NULL,
            name varchar(50),
            bidder varchar(20) NOT NULL,
            bid_amount text,
            PRIMARY KEY (id,name,bidder)
        )""")
        try:
            self.curr.execute("""select id from auction_tb order by id desc limit 1""")
            last_id = self.curr.fetchone()
            self.id_no = last_id[0]
        except:
            self.id_no = 0
        f = open("/opt/lampp/htdocs/scrapy/result.txt", "w")
        f.write("scraping")
        f.close()
        self.clear_ended_item() 
         '''

in the code i've changed the driver directory to match ubuntu (change xampp to lampp and the directories prior) and i've added a placeholder for the website scraping. This part of the script and the whole script runs on windows perfectly. things i've tried:

replacing the .exe file with the equivalent linux file.
loading wine of the computer to execute the .exe when it's called. some things about that... after installing wine i right clicked the executable to open it with wine and wine was not an available option and there was no availability to search the computer for it.
I have seen and tried the linux server options from here How to implement chromedriver in selenium in Linux platform
scrolling down 3/4 down this page they've coded in Javascript something that apparently works for java.. here i'm unsure how to edit my python code in a similar way.https://tecadmin.net/setup-selenium-chromedriver-on-ubuntu/
I change the pathing and do research but mostly i think the problem here is that i don't know how to change my code to make it work for linux. UPDATE*** as suggested by Yosuva A. i used chmod +x in the file directory on chromedriver to make the file executable if it weren't already and then i double clicked the file to see if it would execute, recieved this error message

could not display "chromedriver".

there is no application installed for "shared library" files. do you want to search for an application to open this file?

the file type is : shared library(application/x-sharedlib) allow executing is now checked in properties

UPDATE#2: As suggested by Yosuva A. i used a sample script from chromedriver (chromedriver.chromium.org/getting-started) and got a different error but this clearly says it is finding the chromedriver file.

selenium.common.exceptions.SessionNotCreatedException: Message: session not >created: This version of ChromeDriver only supports Chrome version 78

i am searching for a way to find which chrome i have and downloading the chromdriver to match it. An alternative would be to download chrome version 78. it throws a different error as before

update #3: the sample chrome script now works, trying the same coding line with the original script file

update#4(10 minutes later): Sample script works. Project script fails to operate -gaining error message

Blockquote selenium.common.exceptions.WebDriverException: >> Message: 'chromedriver' executable needs to be in the PATH.

update#5: shows command prompt during running scrapy crawl.

2019-10-10 00:28:29 [py.warnings] WARNING: 

/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/spiderloader.py:37: UserWarning: There are several spiders with the same name:

      Furgeson named 'auction' (in auctionscraping.spiders.auction_spyder)
      Hemingway named 'auction' (in auctionscraping.spiders.auction_spyder)
      Jeffersons named 'auction' (in auctionscraping.spiders.auction_spyder)
      End named 'auction' (in auctionscraping.spiders.auction_spyder)

      This can cause unexpected behavior.
      warnings.warn(msg, UserWarning)

    2019-10-10 00:28:29 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'auctionscraping', 'NEWSPIDER_MODULE': 'auctionscraping.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['auctionscraping.spiders']}
    2019-10-10 00:28:29 [scrapy.extensions.telnet] INFO: Telnet Password: cef133d6b6822838
    2019-10-10 00:28:29 [scrapy.middleware] INFO: Enabled extensions:
    ['scrapy.extensions.corestats.CoreStats',
     'scrapy.extensions.telnet.TelnetConsole',
     'scrapy.extensions.memusage.MemoryUsage',
     'scrapy.extensions.logstats.LogStats']



 Note: error message comes up after the script has failed to run.. when i CTRL + C close the program this error:

> File "/home/disco/.conda/envs/success/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/cmdline.py", line 149, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/crawler.py", line 251, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/crawler.py", line 137, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/crawler.py", line 338, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/spiderloader.py", line 61, in from_settings
    return cls(settings)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/spiderloader.py", line 25, in __init__
    self._load_all_spiders()
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
    for module in walk_modules(name):
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
    submod = import_module(fullpath)
  File "/home/disco/.conda/envs/success/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/lampp/htdocs/scrapy/auctionscraping/auctionscraping/spiders/auction_spyder.py", line 576, in <module>
    f = open("opt/lampp/htdocs/scrapy/result.txt", "w")
FileNotFoundError: [Errno 2] No such file or directory: 'opt/lampp/htdocs/scrapy/result.txt'
Unhandled error in Deferred:
2019-10-10 00:44:07 [twisted] CRITICAL: Unhandled error in Deferred:

>Traceback (most recent call last):
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/crawler.py", line 172, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/crawler.py", line 176, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/opt/lampp/htdocs/scrapy/auctionscraping/auctionscraping/spiders/auction_spyder.py", line 571, in crawl
    yield runner.crawl(Jefferson)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/crawler.py", line 79, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/crawler.py", line 102, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/scrapy/spiders/__init__.py", line 51, in from_crawler
    spider = cls(*args, **kwargs)
  File "/opt/lampp/htdocs/scrapy/auctionscraping/auctionscraping/spiders/auction_spyder.py", line 408, in __init__
    self.driver = webdriver.Chrome('opt/lampp/htdocs/scrapy/chromedriver')
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
    self.service.start()
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 83, in start
    os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
2019-10-10 00:44:07 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/home/disco/.conda/envs/success/lib/python3.7/site-packages/selenium/webdriver/common/service.py", line 76, in start
    stdin=PIPE)
  File "/home/disco/.conda/envs/success/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/home/disco/.conda/envs/success/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'opt/lampp/htdocs/scrapy/chromedriver': 'opt/lampp/htdocs/scrapy/chromedriver'

update#6: i saw there was an error for the .txt file and i've changed locations for that file and redirected it in the script. i am doing that next for the chromedriver file.

update#7: the chromedriver location was supposed to be /opt/lampp... and it was incorrectly marked as opt/lampp.. -- with the changes the browser opens but the script does not run. Plain blank page.

Now i think the chromedriver issue has been resolved and it was mostly a directory inconvenience as the directory was listed several times in the file and initially i only saw the one line in the script where it marked the chromedriver directory. After using ctrl +F with the line listed as the old directory i found all other codelines changed them and it works.

yes, i did change the chromedriver to use the linux version of chromedriver instead of windows' .exe. Do you mean "activating" the linux executable? by properties? I have just placed the linux version in the same directory. — EmpireLearning, Oct 07 '19 at 17:55
Yes I used run chmod +x chromedriver path to make chromedriver as executable — Yosuva Arulanthu, Oct 07 '19 at 18:32
Also what is happening if try to open chromedriver file on your Ubuntu machine? is it opening as normal or any error? — Yosuva Arulanthu, Oct 07 '19 at 19:51
thank you for the reply, it has been helpful to finding a solution. Question is updated, i'm searching for solution to the shared files application. — EmpireLearning, Oct 08 '19 at 05:21
Please check this question https://stackoverflow.com/q/7023942/4595847 — Yosuva Arulanthu, Oct 08 '19 at 06:05
checked this, doesn't seem to answer my question for the execution of chromedriver here — EmpireLearning, Oct 08 '19 at 07:22
You mentioned that you couldn't edit the Java code similar to python. I can help you edit. Can you tell me which code you're trying to convert to Java? — Yosuva Arulanthu, Oct 08 '19 at 07:46
that's an interesting idea but i would like to keep it all one language (python) so not to confuse later on. I need to rewrite the chromedriver python statement to work in a different way. This is the only portion of the code that needs to be changed. — EmpireLearning, Oct 08 '19 at 08:07
Can you try simple python program to test this issue? So that we can find where the issue is. You can find simple selenium python code here https://chromedriver.chromium.org/getting-started — Yosuva Arulanthu, Oct 08 '19 at 12:16
Hey running a simple script has gotten this to actually work. It throws another error, adding that error to the update above — EmpireLearning, Oct 09 '19 at 07:23
Check the chrome version and download the required chromedriver. — Yosuva Arulanthu, Oct 09 '19 at 07:34
You can find the chrome version by clicking menu->help->about — Yosuva Arulanthu, Oct 09 '19 at 07:37
Regarding update 4: are you getting error with sample program or with your program? — Yosuva Arulanthu, Oct 09 '19 at 21:04
I am getting errors with my program. for clarity, ive edited that update. — EmpireLearning, Oct 10 '19 at 06:46
Which means problem with your program. Can you please check? — Yosuva Arulanthu, Oct 10 '19 at 06:48
i am posting cmd when scripts runs - see edit in a few moments — EmpireLearning, Oct 10 '19 at 07:27

EmpireLearning · Answer 1 · 2019-10-10T08:23:35.777

There were three main things with getting chromedriver to work:

change all old directories to the chromedriver to the new directory, do a global search on the text file using CTRL +F. The important part of doing this is when i got an error for chromedriver it didn't show which line it printed error on so where chromedriver was called again with the old directory it got overlooked when changing to the new directory but the prompt still shows that the chromedriver file couldnt be found, the other 2 times chromedriver were called the directory was set as the old directory-- throwing the same error.
Mchromedriver version is the same as chrome version. This was found, among any other external errors, while running a sample script from the chromedriver tutorial page, use programs chromedriver file location to check if the file is unfindeable while using a clean script (tutorial chromedriver script).
Download chromedriver for whichever operating system you're now using and mark the file as executable, install in any directory, just making sure to change your directory when you do.

yes! fixed the chromedriver! The program is broken but it's no longer chromedriver. — EmpireLearning, Oct 11 '19 at 08:34

how do you fix the executable error with chromedriver Windows >>> in a transition to Ubuntu?

1 Answers1