0

I am scraping job posting data from a website using BeautifulSoup. I have working code that does what I need, but it only scrapes the first page of job postings. I am having trouble figuring out how to iteratively update the url to scrape each page. I am new to Python and have looked at a few different solutions to similar questions, but have not figured out how to apply them to my particular url. I think I need to iteratively update the url or somehow click the next button and then loop my existing code through each page. I appreciate any solutions.

url: https://jobs.utcaerospacesystems.com/search-jobs

Christian
  • 7
  • 3
  • 1
    open the browser's developer console's network tab, and you'll see it's sending ajax requests in the background – Fabricator Sep 20 '17 at 23:08

1 Answers1

0

First, BeautifulSoup doesn't have anything to do with GETing web pages - you get the webpage yourself, then feed it to bs4 for processing.

The problem with the page you linked is that it's javascript - it only renders correctly in a browser (or any other javascript VM).

@Fabricator is on the right track - you'll need to watch the developer console and see what the ajax requests the js is sending to the server. In this case, also take a look at the query string params, which include a param called CurrentPage - that's probably the one you want to focus on.

Danielle M.
  • 3,607
  • 1
  • 14
  • 31
  • Thanks... I do not believe my coding knowledge is advanced enough to accomplish this, but hey! At least I scraped what I need from the first page (haha) – Christian Sep 21 '17 at 17:26