0

I have a mini project where there is a list of URLs on the first page and then I have to follow each URL in these list of URLs and open each URL with SplashRequest because I need the returned page to be rendered along with its JavaScript component.

Now, I'm very new to all of these web scraping and scrapy-splash but basically I'm currently stuck because I'm trying to figure out how to pass a variable to the callback function when using SplashRequest. Basically, I have no idea how to pass a variable to our callback function below:

  yield SplashRequest(url=someUrl,callback=theCallBackFunction,args={
               //some args
            })

I know that for normal Scrapy without Splash, we can just say

yield response.follow(url=someUrl,callback=theCallBackFunction,meta={
     //some variables we want to pass to callback function
}

but I don't know how to do it with Scrapy-Splash.

So yeah, looking for help from anyone who's more experience in this area.

Thanks in advance!

blue2609
  • 841
  • 2
  • 10
  • 25
  • Have been answered here: https://stackoverflow.com/questions/32252201/passing-a-argument-to-a-callback-function – Igris Feb 08 '22 at 00:06

1 Answers1

0

Found the answer to this one myself, apparently the SplashRequest also takes meta as its argument just like response.follow so the mechanism of passing variables to the callback function is exactly the same as using a normal scrapy.

blue2609
  • 841
  • 2
  • 10
  • 25