How to run a Selenium program on a VDS (Ubuntu) server?

Question

I wrote a Python program for a specific channel. It's designed to post hentai content to that channel. The program only works with Selenium, but since I developed it on Windows, I'm not sure how to run it on a VDS server. I downloaded an older version of Google Chrome (version 114.0.5735.90) because Selenium doesn't yet support the latest version. Here's the code of the program:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from random import choice, randint
import logging
from aiogram import Bot, Dispatcher, types
from aiogram.contrib.fsm_storage.memory import MemoryStorage

import requests

from os import remove, getcwd

from config import url

TOKEN = "6208140357:AAEB94wTzQGqIITJMyv-GbvWA3OMyZ59Bzg"

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

bot = Bot(token=TOKEN)
storage = MemoryStorage()
dp = Dispatcher(bot, storage=storage)

all_href = []
photos_links = []
src_link = []
group_urls = []

rn = randint(1, 3)

if rn == 1:
    tags = ['amber_%28genshin_impact%29+#', 'lisa_%28genshin_impact%29+', 'barbara_%28genshin_impact%29+', 
        'noelle_%28genshin_impact%29+', 'fischl+', 'sucrose_%28genshin_impact%29+', '%09Beidou', 'Candace', 
        'collei+', 'dehya+', 'eula_%28genshin_impact%29+', 'eula+', 'faruzan+', 'ganyu+', 'hu_tao+', 
        'jean_%28genshin_impact%29+', 'kamisato_ayaka_%28genshin_impact%29+', 'keqing_%28genshin_impact%29+', 
        'kirara_%28genshin_impact%29+', 'kujou_sara+', 'kuki_shinobu+', 'layla_%28genshin_impact%29+',
        'mona_megistus+', 'nilou+', 'ningguang+', 'raiden_shogun+', 'rosaria_%28genshin_impact%29+',
        'sangonomiya_kokomi+', 'shenhe_%28genshin_impact%29+', 'xiangling+', 'yae_miko_%28genshin_impact%29+',
        'yanfei+', 'yelan+', 'yoimiya+']
elif rn == 2:
    tags = ['asta_%28honkai%3A_star_rail%29+', 'fu_xuan', 'herta', 'himeko+', 'kafka_%28honkai%3A_star_rail%29+', 
            'march_7th+', 'natasha_%28honkai%3A_star_rail%29+', 'pela', 'qingque_%28honkai%3A_star_rail%29+', 
            'seele_%28honkai%3A_star_rail%29+', 'serval', 'silver_wolf_%28honkai%3A_star_rail%29+', 'tingyun']
elif rn == 3:
    tags = ['azur_lane',
            'no_bra', 'nipples', 'open_shirt', 'pantsu', 'skirt_lift', 'dress', 'animal_ears', 'bunny_ears',
            'ass', 'bikini', 'see_through', 'seifuku', 'nopan', 'feet', 'cleavage', 'wet']

'''
tags = ['amber_%28genshin_impact%29+#', 'lisa_%28genshin_impact%29+', 'barbara_%28genshin_impact%29+', 
        'noelle_%28genshin_impact%29+', 'fischl+', 'sucrose_%28genshin_impact%29+', '%09Beidou', 'Candace', 
        'collei+', 'dehya+', 'eula_%28genshin_impact%29+', 'eula+', 'faruzan+', 'ganyu+', 'hu_tao+', 
        'jean_%28genshin_impact%29+', 'kamisato_ayaka_%28genshin_impact%29+', 'keqing_%28genshin_impact%29+', 
        'kirara_%28genshin_impact%29+', 'kujou_sara+', 'kuki_shinobu+', 'layla_%28genshin_impact%29+',
        'mona_megistus+', 'nilou+', 'ningguang+', 'raiden_shogun+', 'rosaria_%28genshin_impact%29+',
        'sangonomiya_kokomi+', 'shenhe_%28genshin_impact%29+', 'xiangling+', 'yae_miko_%28genshin_impact%29+',
        'yanfei+', 'yelan+', 'yoimiya+', 'yun_jin+', 'kafka_%28honkai%3A_star_rail%29+', 'herta+', 'azur_lane',
        'no_bra', 'nipples', 'open_shirt', 'pantsu', 'skirt_lift', 'dress', 'animal_ears', 'bunny_ears',
        'ass', 'bikini', 'see_through', 'seifuku', 'nopan', 'feet', 'cleavage', 'wet', 
        'asta_%28honkai%3A_star_rail%29+', 'fu_xuan', 'herta', 'himeko+', 'kafka_%28honkai%3A_star_rail%29+', 
        'march_7th+', 'natasha_%28honkai%3A_star_rail%29+', 'pela', 'qingque_%28honkai%3A_star_rail%29+', 
        'seele_%28honkai%3A_star_rail%29+', 'serval', 'silver_wolf_%28honkai%3A_star_rail%29+', 'tingyun']
'''

t = None
number = 0
number_2 = 0

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-extensions')

webdriver_service = Service("root/kitsu-api/chromedriver_linux64/chromedriver")
driver = webdriver.Chrome(service=webdriver_service, options=chrome_options)

driver.get(f"{url}?page=1&tags={choice(tags)}")

links = driver.find_elements(By.CSS_SELECTOR, 'a.thumb')

for link in links:
    href = link.get_attribute('href')
    all_href.append(href)

while number <= 5:
    if number_2 >= 15:
        number += 5
        print("всі фото на цьому сайті вже були опубліковані...")

    with open("all_photos.txt", "a+", encoding='utf-8') as f:
        lines = f.readlines()

        link_3 = choice(all_href)

        t = None 

        for i in lines:
            line = lines[i]
            if link_3 in line:
                t = True
                break 

        if t == True:
            number_2 += 1
            continue
        else:
            photos_links.append(link_3)
            f.write(f"{link_3}\n")
            number += 1

driver.quit()

if number >= 5:
    for i in range(6):
        driver = webdriver.Chrome(service=webdriver_service, options=chrome_options)

        driver.get(photos_links[i])

        links = driver.find_elements(By.CSS_SELECTOR, 'img.image')

        for link in links:
            image = link.get_attribute('src')
            src_link.append(image)

        driver.quit()

    @dp.message_handler(commands=['start'])
    async def start(message: types.Message):
        user_id = 2139742877

        if int(user_id) != 2139742877:
            await message.reply("нажаль, цього бота може використовувати тільки його розробник. Пишіть сюди: @duxtaa")
        else:
            await message.reply("Bot is running")

    @dp.message_handler(commands=['chat_id'])
    async def chat_id_command(message: types.Message):
        chat_id = message.chat.id
        await message.answer(chat_id)

    @dp.message_handler(commands=['image'])
    async def send_photos(message: types.Message):
        user_id = 2139742877

        if int(user_id) != 2139742877:
            await message.reply("нажаль, цього бота може використовувати тільки його розробник. Пишіть сюди: @duxtaa")
        else:
            for i in range(1, 6):  
                url = src_link[i]
                response = requests.get(url)
                photo = response.content
                with open(f"photo_{i}.jpg", "wb") as file:
                    file.write(photo)

                with open(f"photo_{i}.jpg", "rb") as file:
                    await bot.send_document(-1001919858113, document=file, disable_notification=True)

                group_urls.append(url)

            message_id = message.message_id
            print(message_id)

            photo_ids = group_urls
            await bot.send_media_group(chat_id=-1001919858113, media=[types.InputMediaPhoto(media=photo_id) for photo_id in photo_ids])
            await bot.send_message(chat_id=-1001919858113, text=f"{rn}")

            for i in range(1, 6):
                remove(f"photo_{i}.jpg") 

            await bot.stop_polling()

    async def on_startup(dp):
        await bot.send_message(chat_id=-1001919858113, text="Bot is running")
        await send_photos(types.Message())

    if __name__ == '__main__':
        from aiogram import executor
        executor.start_polling(dp, skip_updates=True, on_startup=on_startup)

The program was running through some error that I couldn't resolve. Here's the error itself:

Traceback (most recent call last):
File "/root/kitsu-api/send-anime-photos.py", line 7, in <module>
driver=webdriver. Chrome (options=options)
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py",
super (). __init__(
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py",
super ().__init__(
line 54, in
_init_
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py",
line 206, in init
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 291, in start_session
response = self.execute (Command.NEW_SESSION, caps) ["value"]
File "/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py", line 346, in execute
"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py", line 245, in check_response
File
raise exception_class (message, screen, stacktrace)
selenium.common.exceptions.WebDriverException:
Message: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:
self.start_session (capabilities)
self.error_handler.check_response (response)
#0 0x564f0ab2d4e3 <unknown>
#1 0x564f0a85cc76 <unknown>
#2 0x564f0a885d78 <unknown>
#3 0x564f0a882029 <unknown>
#4 0x564f0a8c0ccc <unknown>
#5 0x564f0a8c047f <unknown>
#6 0x564f0a8b7de3 <unknown>
# 7 0x564f0a88d2dd <unknown>
#8 0x564f0a88e34e <unknown>
#9 0x564f0aaed3e4 <unknown>
# 10 0x564f0aaf13d7 <unknown>
#11 0x564f0aafbb20 <unknown>
#12 0x564f0aaf2023 <unknown>
#13 0x564f0aac01aa <unknown>
#14 0x564f0ab166b8 <unknown>
#15 0x564f0ab16847 <unknown>

The server itself runs on Ubuntu version 22.

Please add the selenium log error. – Jortega Jul 24 '23 at 20:54 — Jortega, Jul 24 '23 at 20:54
I think the file is called chromedriver.log – Jortega Jul 24 '23 at 21:00 — Jortega, Jul 24 '23 at 21:00

undetected Selenium · Answer 1 · 2023-07-24T21:25:43.260

This error message...

selenium.common.exceptions.WebDriverException:
Message: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

...implies that google-chrome didn't start or crashed immediately and this often happens when running ChromeDriver / Chrome using a special test harness (e.g. an IDE) or continuous build system (e.g. Jenkins).

Solution

A couple of debugging steps:

Try to launch the same Chrome binary that your test uses from a normal user command prompt (check your chromedriver.log file to see what Chrome is being used).
Incase you are passing any special command line switches/arguments to Chrome, ensure to include those too. If Chrome still fails to start correctly, you may need to fix your installation of Chrome by re-installing.
If you are able to run Chrome from a command prompt, the next step should be to test if the same problem occurs when running your test in a simple testing environment (preferably launching the test binary/script directly from a normal user's command prompt).
You also need to check that you are able to launch Chrome directly from your test (without using WebDriver/ChromeDriver).

However a common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and highly discouraged. Please configure your environment to run Chrome as a regular user instead.

References

You can find a relevant detailed discussion in:

How to check all these points? (I apologize for such stupid questions) — Duxtaa, Jul 25 '23 at 19:58

How to run a Selenium program on a VDS (Ubuntu) server?

1 Answers1

Solution

References