-1

I realize this is a relatively simple question but I haven't found the answer yet.

I'm using driver.get() in a for loop that iterates through some urls. To help avoid my IP address getting blocked, I've implemented time.sleep(5) before the driver.get statement in the for loop.

Basically, I just want a wait period to make my scraping seem more natural.

I think time.sleep may be causing page crashes. What is the equivalent of time.sleep in selenium? From what I understand, implicitly_wait just sets the amount of time before throwing an exception, but I'm not sure that's what I want here? I want a specific amount of time for the driver to wait.

halfer
  • 19,824
  • 17
  • 99
  • 186
DiamondJoe12
  • 1,879
  • 7
  • 33
  • 81
  • `time. sleep()` is from `Python` however Selenium does have the `.pause` method and that is for web element interaction load time. However, I don't think that you need `.pause` here. Your question is not really clear to me. Did you intend to ask why the page crashes during Selenium code execution? – cruisepandey Mar 19 '22 at 18:03
  • Note that we prefer a technical style of writing here. We gently discourage greetings, hope-you-can-helps, thanks, advance thanks, notes of appreciation, regards, kind regards, signatures, please-can-you-helps, chatty material and abbreviated txtspk, pleading, how long you've been stuck, voting advice, meta commentary, etc. Just explain your problem, and show what you've tried, what you expected, and what actually happened. – halfer Mar 20 '22 at 00:20

2 Answers2

2

time.sleep()

The sleep() function is from the time module which suspends execution of the current thread for a given number of seconds.

Now, WebDriver being a out-of-process library which instructs the browser what to perform and at the same time the web browser being asynchronous in nature, WebDriver can't track the active, real-time state of the HTML DOM. This gives rise to some intermittent issues that arise from usage of Selenium and WebDriver those are subjected to race conditions that occur between the browser and the user’s instructions.

As of now Selenium doesn't have any identical method to time.sleep(), however there are two equavalent methods at your disposal and can be used as per the prevailing condition of your automated tests.

  • Implicit wait: In this case, WebDriver polls the DOM for a certain duration when trying to find any element. This can be useful when certain elements on the webpage are not available immediately and need some time to load.

    def implicitly_wait(self, time_to_wait) -> None:
    """
    Sets a sticky timeout to implicitly wait for an element to be found,
       or a command to complete. This method only needs to be called one
       time per session. To set the timeout for calls to
       execute_async_script, see set_script_timeout.

    :Args:
     - time_to_wait: Amount of time to wait (in seconds)

    :Usage:
        ::

            driver.implicitly_wait(30)
  """
  self.execute(Command.SET_TIMEOUTS, {
      'implicit': int(float(time_to_wait) * 1000)})

  • Explicit wait: This type of wait allows your code to halt program execution, or freeze the thread, until the condition you pass it resolves. As an example:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
0

There is no specific method in Selenium for hardcoded pauses like a time.sleep() general Python method.
As you mentioned there is an implicitly_wait and Expected Conditions explicit WebDriverWait waits but both these are NOT a hardcoded pauses.
Both the implicitly_wait WebDriverWait are used for setting the timeout - how long time to poll for some element presence or condition, so if that condition is fulfilled or the element is presented the program flow will immediately continue to the next code line.
So, if you want to put a pause you have to use some general Python method that will suspend the program / thread run like the time.sleep().

Prophet
  • 32,350
  • 22
  • 54
  • 79
  • Not sure if you are aware of `pause` method in selenium, it is equivalent to `time.sleep()` that is available in Python. – cruisepandey Mar 19 '22 at 17:59
  • I indeed not familiar with such Selenium method. Maybe because I mostly use Selenium with Java? So, you think I should remove this answer @cruisepandey? – Prophet Mar 19 '22 at 18:26
  • No that's not I meant. You can keep the answer, if OP is fine with the solution I don't have any issue. In Java also when you create an actions class object, you'd see .pause method. – cruisepandey Mar 19 '22 at 18:37
  • OK, I thought about Selenium itself :) – Prophet Mar 19 '22 at 18:39