Crawl url, search and save results - no specific address

Asked Sep 06 '18 at 16:12

Active Sep 06 '18 at 16:12

Viewed 104 times

The page I am trying to crawl is a directory of chiropractors. I set my search criteria and get search result but the address stays the same. It has no element of search, page number or anything so I am not able to crawl it the usual way.

Is there any way to crawl to a page, set criteria for search and save results?

Thank you.

asked Sep 06 '18 at 16:12

fatima

1

is it AJAX page? please share the url – Sohan Das Sep 06 '18 at 16:57
1

You can use a library like [selenium](https://www.seleniumhq.org/) to interact with the page and pull the source at any given time – wpercy Sep 06 '18 at 20:02
@ wpercy Yes, I have been trying to use selenium but it is all new to me so I am adding more questions on SO. Thank you. – fatima Sep 06 '18 at 23:12
@ Sohan Das - https://www.acatoday.org/Find-a-Doctor – fatima Sep 06 '18 at 23:13
you have to use selenium. – Sohan Das Sep 07 '18 at 11:56
Please [edit] the question to give some example search criteria and the results you are trying to extract. – Martin Evans Sep 08 '18 at 12:35
If you want a tool that can be used graphically and scripted then you can use OWASP ZAP. The ZAP ajax spider uses selenium under the hood but you wont need to code anything. – Simon Bennetts Sep 11 '18 at 07:35

Crawl url, search and save results - no specific address

0 Answers0