0

Javascript is hiding some elements but when I am scraping the data using scrapy those elements are scraped as if they are not hidden by Javascript.

Link 1: http://www.limeroad.com/turquoise-sling-ivory-tag-p38444 Link 2: http://www.limeroad.com/baby-pink-wallet-eske-p35768#pOverlay=none

For Example: When I crawled these two links using Scrapy with xpath (//div[@id='productDetails']/ul[@class='details']/li[@id='add_to_cart_li']/a[@id='add_to_cart']/span[@class='buy-txt']), I got "Buy now" as a text in both the links even though one link doesnot contain "buy now" Text. This is happening because Javascript is changing the display . Scrapy doesn't load the Javascript changed content.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
cdev
  • 93
  • 8

1 Answers1

0

Not sure how the elements are hidden. see. What is the difference between visibility:hidden and display:none?

If you are using xpath to extract the data then something like this(http://www.w3schools.com/css/css_display_visibility.asp) would help. Mind you passing the link to give an idea how to use xpath for visible elements.

Hope this is of any help.

Community
  • 1
  • 1
Biswanath
  • 9,075
  • 12
  • 44
  • 58