-1

I am a beginner in programming and I am trying to make a scraper. As of right now I'm using the requests library and BeautifulSoup. I provide the program a link and I am able to extract any information I want from that single web page. What I am trying to accomplish is as follows... I want to provide a web page to the program, the web page that I provide is a search result where there is a list of links that could be clicked. I want the program to be able to get the links of those search results, and then scrape some information from each of those specific pages from the main web page that I provide.

If anyone can give me some sort of guidance on how I could achieve this I would appreciate it greatly! Are there some other libraries I should be using? Is there some reading material you could refer me to, maybe a video?

Student
  • 13
  • 3
  • 2
    Hello, Student. Stack Overflow is more for helping people with fixing the problems in their code than for code suggestions. Please try to instead put down what code you currently have, and what problems are being faced for questions in the future. If you need guidance, you can also take a look at the questions already posted on the site. ||| In regards to your question, what you are looking for is probably the urllib requests library, take a look at this question: https://stackoverflow.com/questions/1843422/get-webpage-contents-with-python – Enthus3d Feb 18 '20 at 18:39
  • 1
    Hi, thank you for your the reply! I will look more into urllib! – Student Feb 18 '20 at 23:51
  • That's great! If you run into any specific problems while implementing the scraping, you can ask for help on the site. – Enthus3d Feb 19 '20 at 15:22

1 Answers1

0

You can put all the url links in a list then have your request-sending function loop through it. Use the requests or urllib package for this.

For the search logic, you would want to look for the <a> tag with href property.

isopach
  • 1,783
  • 7
  • 31
  • 43