1

I need to crawl websites and extract some information from dynamically created pages after a form submission.

The information which I need to crawl would mostly come from databases on these sites.

Added:

Crawlers usually work by jumping from one hyper-link to another. So these are mostly static pages. What about crawling pages that are not statically present but created on the fly.

AJ.
  • 2,561
  • 9
  • 46
  • 81
  • That's nice, can you provide more info? – leppie Dec 14 '10 at 11:23
  • I also have same Problem. how can get dynamic website. http://stackoverflow.com/questions/28085211/any-possiable-to-crawl-open-web-browser-data-using-aperture – BasK Jan 22 '15 at 12:05

1 Answers1

1

From crawler's point of view there's no big difference. You're still getting genrated HTML.

The only thing you need to be careful about is links leading to infinite number of pages, e.g. calendar that's dynamically generated and has links to next/previous month/year.

Kornel
  • 97,764
  • 37
  • 219
  • 309