How can I get all the links on a site without using the html code?

Question

Using python 3. I've been trying to find a way to get links that are on dynamic pages but I'm only getting things in the href and the links I need aren't there.

For example, trying to get the link to a specific tournament from this page, but they don't appear in the page source. Is this something I can even do with BeautifulSoup or Selenium?

Thanks!

Hi, Can you post what you have tried. Maybe an example output would be nice as well. — Arundeep Chohan, Dec 18 '20 at 06:48
Maybe the links are not in the page source, but if you inspect the webpage you mentioned, there are tags with the relative link to all tournaments. To get the DOM of the page you should use selenium, not BS. — CarlosSR, Dec 18 '20 at 09:29

score 1 · Answer 1 · answered Dec 18 '20 at 06:45

go to the page and look in :

<section class="page-section">
<div class="page-section__inner">
<table class="table">
<tbody>
<tr class="whatson-table__tournament>"
...

There is everything you need to build the links yourself.
you won't find the complete URL since they use a JS link builder or Server side controllers.

score 1 · Answer 2 · answered Dec 18 '20 at 06:48

I had a look at that page and the links are href. BUT, there are other ways that links can work, one can use javascript to also create links. I found this on stackoverflow just to show that it could be difficult to get all the "links":

Javascript: Setting location.href versus location

Also, see this: https://www.w3schools.com/js/js_window_location.asp

score 0 · Answer 3 · answered Dec 18 '20 at 06:48

0

Try to use parametrized xpath for e.g for your table //tbody/tr[1]/td[1]/a[1] and enstead tr[1] use tr["i"] where i is iterator in loop. First you will need to get num of rows like //tbody/tr. This will help you fore the tournaments links.

answered Dec 18 '20 at 06:48

Gaj Julije

1,538
15
32

How can I get all the links on a site without using the html code?

3 Answers3