Is there any way to find the URL that you are currently scraping?

Question

I'm currently trying to create a spider which crawls each result and takes some info from each of them. The only problem is that I don't know how to find the URL that I'm currently on (I need to retrieve that too).

Is there any way to do that?

I know how to do that using Selenium and Scrapy-Selenium, but I'm only using a simple CrawlSpider for this project.

current_url = response.request.url – User One Sep 05 '21 at 14:31 — User One, Sep 05 '21 at 14:31
@UserOne can you add that as an answer, so I can accept it? – Jonathan Simpson Sep 05 '21 at 15:46 — Jonathan Simpson, Sep 05 '21 at 15:46

score 0 · Accepted Answer · answered Sep 05 '21 at 15:56

0

You can use:

current_url = response.request.url

answered Sep 05 '21 at 15:56

User One

287
2
9

Is there any way to find the URL that you are currently scraping?

1 Answers1