How to get all URLs in a Wikipedia page

Question

It seems like Wikipedia API's definition of a link is different from URL? I'm trying to use the API to return all the urls in a specific wiki page.

I have been playing around with this query that I found from this page under generators and redirects.

score 16 · Accepted Answer · answered Feb 14 '13 at 21:37

I'm not sure why exactly are you confused (it would help if you explained that), but I'm quite sure that query is not what you want. It lists links (prop=links) on pages that are linked (generator=links) from the page “Title” (titles=Title). It also lists only the first page of links on the first page of links (with page size the tiny default value of 10).

If you want to get all the links on the page “Title”:

Use just prop=links, you don't want the generator.
Increase the limit to the maximum possible by adding pllimit=max (pl is the “prefix” for links)
Use the value given in the query-continue element to get to the second (and following) page of results.

So, the query for the first page would be:

http://en.wikipedia.org/w/api.php?action=query&titles=Title&prop=links&pllimit=max

And the second (and in this case, final) page:

http://en.wikipedia.org/w/api.php?action=query&titles=Title&prop=links&pllimit=max&plcontinue=226160|0|Lieutenant_General

Another thing that might be confusing you is that links returns only internal links (to other Wikipedia pages). To get external links, use prop=extlinks. You can also combine the two into one query:

http://en.wikipedia.org/w/api.php?action=query&titles=Title&prop=links|extlinks

Thanks for the answer! I think i was confused cause i didn't think of adjusting the query limit (silly of me..) so it was only showing first several. Also, the link text and the appeared text on the page are often different so i wasn't able to verify if those links were indeed the ones on the page... (some were, some weren't so i was confused as to what i was actually getting..) Thanks again! — gchao, Feb 14 '13 at 22:59

Bill the Lizard · Answer 2 · 2019-09-18T00:12:25.597

Here's a Python solution that gets (and prints) all the pages linked to from a particular page. It gets the maximum number of links in the first request, then looks to see if the returned JSON object has a "continue" property. If it does, it adds the "plcontinue" value to the params dictionary and makes another request. (The last page of results returned will not have this property.)

import requests

session = requests.Session()

url = "https://en.wikipedia.org/w/api.php"
params = {
    "action": "query",
    "format": "json",
    "titles": "Albert Einstein",
    "prop": "links",
    "pllimit": "max"
}

response = session.get(url=url, params=params)
data = response.json()
pages = data["query"]["pages"]

pg_count = 1
page_titles = []

print("Page %d" % pg_count)
for key, val in pages.items():
    for link in val["links"]:
        print(link["title"])
        page_titles.append(link["title"])

while "continue" in data:
    plcontinue = data["continue"]["plcontinue"]
    params["plcontinue"] = plcontinue

    response = session.get(url=url, params=params)
    data = response.json()
    pages = data["query"]["pages"]

    pg_count += 1

    print("\nPage %d" % pg_count)
    for key, val in pages.items():
        for link in val["links"]:
            print(link["title"])
            page_titles.append(link["title"])

print("%d titles found." % len(page_titles))

This code was adapted from the code in the MediaWiki API:Links example.

How to get all URLs in a Wikipedia page

2 Answers2

Linked