0

I'm trying to scrape this website: https://www.ikea.com/mx/es/cat/vajillas-18860/

I have the whole thing working fine, except for one thing that is bugging me.

When I load that website as a normal user with my browser, I always get around 99 products. However, when I load it using chrome webdriver and selenium (without any processing occurring yet), I sometimes get around 68 products, and sometimes I do get the 99 products.

My guess is the site somehow knows I'm using a scraper, and makes some weird stuff. I've tried adding some options I found in other posts to the webdriver, but have had no success yet:

chrome_opt.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_opt.add_experimental_option("useAutomationExtension", False)

Do you guys know what could be causing this, and how to prevent it?

EDIT1

Tried adding different user-agent as suggested, still didn't work:

chrome_opt.add_argument( "user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:63.0) Gecko/20100101 Firefox/63.0" )

99 products 68 products

Alain
  • 339
  • 3
  • 19
  • Have you try different user agents in some sites it is the problem (I didn't check for this site) https://stackoverflow.com/questions/29916054/change-user-agent-for-selenium-web-driver might help – itai Oct 28 '21 at 16:47
  • I tried using this but still no luck (not sure if a particular user-agent would work?): chrome_opt.add_argument( "user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:63.0) Gecko/20100101 Firefox/63.0" ) – Alain Oct 28 '21 at 17:38
  • 1
    I only get 68 results in my regular browser. By the way, why don't you call the api directly? There's no need to scrape – RJ Adriaansen Oct 28 '21 at 18:24
  • There's an IKEA API?! I can't seem to find it... I just found this endpoint for checking availability of products, but it doesn't seem to work: http://www.ikea.com/mx/es/iows/catalog/availability/90451154/ – Alain Oct 28 '21 at 18:55
  • 1
    There's another choice in your second screenshot... "Tamano?" What is that? You may just be seeing a difference due to cookies stored in the manual user session.... something like that, but I'd figure out why there's an extra pulldown there in the webdriver screen. – pcalkins Oct 28 '21 at 19:55
  • Well, i'll be damned... it was cookies... somehow it sometimes got 68 results, and sometimes 99... I've been running circles with this issue for the past day and a half and never thought of cookies. Thanks man! – Alain Oct 28 '21 at 20:56
  • 2
    @Alain there is an api, check [this](https://github.com/vrslev/ikea-api-client). It'll return 68 search results. – RJ Adriaansen Oct 28 '21 at 22:52
  • 1
    What happens if you open the website using an incognito browser? Just to rule out cokes and other things – Taco Verhagen Oct 29 '21 at 19:32

0 Answers0