Scrape data in JS Pop Up Python

Question

I want to know that can we scrape data of a specific field from a pop up generated on a page using Python ? If yes, please suggest.

I am trying to scrape it, but it is not getting detected and return me an empty list. I am using Python and Beautiful soup to do the job .

Have you looked into how the popup is being generated? Is it javascript? If its javascript you probably should use phantomjs as well — joydeep bhattacharjee, Feb 10 '17 at 08:22
Can you extract data with regular expressions from JavaScript that inserts that pop-up? Also, if data in the popup you're trying to extract is retrieved from the HTTP request, you can send that request directly and skip HTML parsing. — ilyazub, Jan 20 '21 at 14:29

score 1 · Answer 1 · edited May 23 '17 at 12:33

1

You can not use scrapy for JS

You can also solve it with ScrapyJS :

This library provides Scrapy+JavaScript integration using Splash.

Follow the installation instructions for Splash and ScrapyJS, READ the answer here

OR

You can use Selenium:

Follow the installation instructions Selenium Doc

edited May 23 '17 at 12:33

Community

answered Feb 10 '17 at 08:41

Karan Chudasama

1 Answers1