1

I'm new to web scraping so I'm fooling around with scrapy and trying to crawl a certain website.

I'm working with the scrapy shell on windows and just trying to establish the proper XPath to a particular element I want to access. The element is a schedule, this is the HTML:

enter image description here

I'm trying to access the rv-schedule-module and all its sub-nodes. I'm able to access all nodes up until the rv-schedule-module however beyond that all XPath calls return null. For instance:

enter image description here

The progression of calls returns data until I want to access a div underneath the rv-schedule-module. That call returns null.

What am I doing wrong?

credo56
  • 423
  • 1
  • 8
  • 19
  • 1
    Maybe that content is dynamically generated? Maybe share the website in question – Rafael Almeida Jun 14 '16 at 02:53
  • https://rutgers.n.rivals.com/ is the site. The element I'm trying to access is the schedule of future games. It's the final element in the right hand column – credo56 Jun 15 '16 at 14:03

1 Answers1

2

Just as I suspected that content is dynamically created because it's handled by javascript!

When you inspect the element it will be there but if you check the page source it won't. Scrapy by itself doesn't handle javascript, you'll need something like scrapy-splash or Selenium.

There is a really good post of the all mighty Alex explaining how to use it - https://stackoverflow.com/a/30378765/2781701

Community
  • 1
  • 1
Rafael Almeida
  • 5,142
  • 2
  • 20
  • 33