3

I am trying to scrape Soundcloud with Scrapy and create a dataframe of artist information.

I tried extracting information of one artist with Xpath manually through terminal. The code below redirects me to an error page. Does Soundcloud do not support scraping? Can you give me advice on how to do this with Scrapy?

scrapy shell https://soundcloud.com/stavroz
view(response)
Denobee6
  • 29
  • 5
  • Probably javascript issues. Scrapy does not interpret javascript natively. Try using scrapy + splash or selenium. – Corentin Limier May 27 '19 at 14:58
  • @CorentinLimier I will look into this thank you very much – Denobee6 May 27 '19 at 15:30
  • See https://stackoverflow.com/q/8550114/939364 – Gallaecio May 28 '19 at 07:20
  • Also doing a `view(response)` will not always work, because inside the browser the page may execute some javascript and change response or load extra info. So its best not to view response directly in browser, rather see the html in the shell – Tarun Lalwani Feb 17 '21 at 07:54

0 Answers0