1

As I learn selenium with python, I find that I may have made a mistake that leads to an exception while I adapt it to the exact flow of the web application. As such, the scripts exits with an error, and python loses control over the driver.

Fixing the mistake takes some time while a new browser is created; that also creates a lot of temporary files that are required for the browser profile (rust_mozprofile). In short, fixing the error loses me significant time.

I'm looking for an answer specifying a workflow with selenium that allows me to reuse the browser instance, make mistakes in syntax, intent or error handling, and correct them without neither reloading the script, respawning the browser, nor redoing all steps in the website.

I'd prefer a Firefox focused solution, but Chrome answers are also welcome.

ferreiradev
  • 327
  • 5
  • 16

1 Answers1

0

Use the VS Code Debugger

Most of the issues mentioned could be avoided by first doing an exhaustive review of the website so that you start coding your script with "perfect" knowledge of the every site transition, delay, button identifier, and so on. Realistically, this would be too much work.

I will provide a simple example of the workflow I use to develop:

Example. Retrieve the first picture from #cats in Instagram

Step 1.

Start with boilerplate code that imports all relevant modules so that if I need them, I can call them immediately. A good start is:

bot_template.py:


    import logging
    import pickle
    import time
    from inspect import getsourcefile
    from os.path import abspath
    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.common.by import By
    from selenium.common.exceptions import NoSuchElementException, TimeoutException
    from selenium.webdriver.remote.webelement import WebElement
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.firefox.options import Options

    driver = webdriver.Firefox()
    # your breakpoint should go in this line
    driver.quit()

At this point try doing the process manually of going to Instagram and getting the first picture in the tags feed. If you did you would find that you can go to https://www.instagram.com/explore/tags/cats/ for the cat tag.

enter image description here

Step 2

Add a breakpoint in the line right after driver = webdriver.Firefox() which spawns Firefox. Breakpoints are added by clicking in the empty region immediately left of the line numbers or with the default shortcut (F9).

Step 3

In VSCode, Press F1 and type Start Debugging, and select Python File, or use the default shortcut (F5).

Step 4

In VSCode, Press F1 and type Focus on Debug Console View. Switch over to the debug console.

Step 5

Make mistakes in the debug console as you try to build your script. You need the bot to go to https://www.instagram.com/explore/tags/cats/, so you write: driver.Get(r'https://www.instagram.com/explore/tags/cats/'). Next thing you see is:

Error: AttributeError: 'WebDriver' object has no attribute 'Get'

Step 6

Correct your mistake. You realize the method is actually called get (lowercase). Type that again in the VSCode Debug Console. The browser will now navigate to the right URL, no need to respawn a new one due to a typo.

>>> driver.get('https://www.instagram.com/explore/tags/cats/')

Step 6

Switch over to the browser you are controlling from Python. You are free to do whatever you want to it, in whatever tempo/order you want. You open the inspector and realize that you can use XPATH in the browser terminal $x('//div[@class="_aabd _aa8k _aanf"]') to get the array of images. Therefore, you type in the VSCode debug console:

>>> image = driver.find_element(By.XPATH, '//div[@class="_aabd _aa8k _aanf"]')

Step 7

You start trying to get the image src. Therefore, you type image.get_attribute('src') in the debug console. It does not work because you selected a div that does have a 'src' html attribute, rookie mistake. The code would have terminated in regular execution and you would have lost control over the browser, but you are on the debugger, so you retain control and your precious time. You keep trying in your debugging playground, until you figure out that you can fix this by changing the xpath. You try typing in the debug console and that gets you the image url.

>>> image = driver.find_element(By.XPATH, '//*[@class="_aabd _aa8k _aanf"]//img[@src]')
>>> src = image.get_attribute('src')

Step 8

Download the image from the url. You keep the browser and vscode session open and go look for ways to download the image. You notice you need other libraries. No problem, try typing the following in same debug console session and you will have your image downloaded.

# Changes the current dir to the project folder and downloads as 'image.png'
>>> import os
>>> import urllib
>>> os.chdir('.')
>>> urllib.urlretrieve(src, 'image.png')

Step 9

Keep building interactively until your bot does all it has to do. Commit the lines of code that gave you the desired result to your file bot_template.py. Once you are done, test the whole thing by running the script without debugging. You never had to respawn another browser session and lose time.

ferreiradev
  • 327
  • 5
  • 16