0

I am using selenium and chrome to search on google. But it is returning different layouts for pagination. I am using different proxies and different user agents using the fake_useragent library.

I only want the second image layout. Does anybody know how can I get it every time?

First Image

Second Image

user8511791
  • 45
  • 1
  • 6

3 Answers3

1

The issue was fake_useragent library was returning old user-agents sometimes even if I update the database. I tried this library(https://pypi.org/project/latest-user-agents/) and it returns newer user-agents.

Here is the working code.

from latest_user_agents import get_latest_user_agents
import random
from selenium import webdriver

PATH = 'C:\Program Files (x86)\chromedriver.exe'

proxy = ''
url = ''

user_agent = random.choice(get_latest_user_agents())

options = webdriver.ChromeOptions()
options.add_argument(f'--proxy-server={proxy}')       
options.add_argument(f'user-agent={user_agent}')   

driver = webdriver.Chrome(PATH, options=options)  

driver.get(url)
user8511791
  • 45
  • 1
  • 6
0

The difference between the two layouts is when you disable javascript, Google will show the pagination as the first image layout.

To ensure that you get the second layout every time, you would need to make sure javascript is enabled.

If you have a chrome driver from selenium like: options = webdriver.ChromeOptions(), the following would make sure javascript is always enabled:

options.add_argument("--enable-javascript")

Edit based on OP's comment

I got it working by using the latest_user_agents library. The fake_useragent library was returning old user-agents sometimes. That's why it was showing the old layout.

Installing the latest_user_agents library: https://pypi.org/project/latest-user-agents/

Parzival
  • 2,051
  • 4
  • 14
  • 32
  • I enabled the javascript but it still sometimes give the first layout. – user8511791 Apr 19 '21 at 20:26
  • @Parzial Thanks for your time. I got it working by using the latest_user_agents library (https://pypi.org/project/latest-user-agents/). The fake_useragent library was returning old user-agents sometimes. That's why it was showing old layout. – user8511791 Apr 20 '21 at 22:14
  • @user8511791 Ah I see. I'm glad you figured it out! I'll add your comment to the answer so it might help future users too. – Parzival Apr 20 '21 at 22:17
0

Hey Dont try to automate google and google products by automation tools because every day google are changing webelements and view of thier pages.

For multiple reasons, logging into sites like Gmail and Facebook using WebDriver is not recommended. Aside from being against the usage terms for these sites (where you risk having the account shut down), it is slow and unreliable.

The ideal practice is to use the APIs that email providers offer, or in the case of Facebook the developer tools service which exposes an API for creating test accounts, friends, and so forth. Although using an API might seem like a bit of extra hard work, you will be paid back in speed, reliability, and stability. The API is also unlikely to change, whereas webpages and HTML locators change often and require you to update your test framework.

Logging in to third-party sites using WebDriver at any point of your test increases the risk of your test failing because it makes your test longer. A general rule of thumb is that longer tests are more fragile and unreliable.

WebDriver implementations that are W3C conformant also annotate the navigator object with a WebDriver property so that Denial of Service attacks can be mitigated.

Justin Lambert
  • 940
  • 1
  • 7
  • 13