1

Does it make a difference whether I load a website every two seconds or always reload it as soon as the website has reloaded? So if I use python and selenium and the page or its subdirectories open every two seconds or as soon as the subdirectories are available? or the server may be overloaded.

for each in self.all:
    self.driver.get(each)

or

WebDriverWait(self.driver, 60).until(
                    EC.presence_of_element_located((By.TAG_NAME, 'p')))
self.driver.get(each)
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
LittleDuck19
  • 47
  • 6
  • after a get method call, the driver will wait until a page load happens. (by default, though you can set PageLoadStrategy to "eager" or "none"...) Your 2nd method just adds a 1/2 second of time to poll... or 60 if it never finds a

    tag.

     – pcalkins Jul 17 '20 at 19:48

2 Answers2

1

Yes, it makes a difference. If you will do many requests in small amount of time, you can receive a 429 status code which means that you make too many requests. Also, you might receive captcha or trigger other security stuff.

Aleksander Ikleiw
  • 2,549
  • 1
  • 8
  • 26
1

As far as Selenium is concerned

  • Selenium automates browsers. That's it!
  • What you do with that power is entirely up to you.

I don't see any issue in either of your code block involving:

  • Iterating through get() requests.
  • Inducing WebDriverWait for presence_of_element_located()

The only grey area is the expected_conditions of presence_of_element_located(). If your usecase is to extract the text you must induce WebDriverWait for visibility_of_element_located()

References

You can find a couple of relevant discussions in:

A word of caution

Do not send automated queries to website which doesn't permits as per their Terms of Service else you may get blocked.

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352