Python - How to get the page Wikipedia will redirect me to?

Question

I want to store a few different Wikipedia links but I don't want to store two different links to the same page twice. For example the following links are different but they point to the same Wikipedia page:

https://en.wikipedia.org/w/index.php?title=(1S)-1-Methyl-2,3,4,9-tetrahydro-1H-pyrido-3,4-b-indole&redirect=no 
https://en.wikipedia.org/w/index.php?title=(1S)-1-methyl-2,3,4,9-tetrahydro-1H-pyrido-3,4-b-indole&redirect=no
__________________________________________________|___________________________________________________________

The only difference is that one uppercase character. Or the following links:

https://en.wikipedia.org/wiki/(0,1)-matrix 
https://en.wikipedia.org/wiki/(0,1)_matrix 
___________________________________|______

That are only different because one has '-' and the other has '_'(' '). So what I want is storing only one of them or the following links:

https://en.wikipedia.org/wiki/Tetrahydroharman 
https://en.wikipedia.org/wiki/Logical_matrix

I have already tried the answer to this SO question. But it didn't work for me. (The result is the initial URL for me, not the one wiki redirects me to in the browser) So how can I achieve what I'm looking for!?

Could you please explain what result you exactly get by trying the solution in the SO question you linked to and which of the solutions you tried exactly!? — yukashima huksay, Nov 28 '17 at 17:41
@aran I used the accepted answer and I already explained what I got. — tgwtdt, Nov 28 '17 at 17:42

score 5 · Answer 1 · edited Oct 18 '22 at 08:37

The MediaWiki API provides various endpoints used in Wikipedia. You can use the MediaWiki Action API to get the target page of a redirect.

the result can be in JSON format (for example)

all you need is to parse it to get the value of the element to or the element title

This query will retrieve the target page for 'Halab':

https://en.wikipedia.org/w/api.php?action=query&titles=Halab&redirects&format=json

Result:

{  
   "batchcomplete":"",
   "query":{  
      "redirects":[  
         {  
            "from":"Halab",
            "to":"Aleppo"
         }
      ],
      "pages":{  
         "159244":{  
            "pageid":159244,
            "ns":0,
            "title":"Aleppo"
         }
      }
   }
}

In Python:

import json
import requests

query = requests.get(r'https://en.wikipedia.org/w/api.php?action=query&titles={}&redirects&format=json'.format('Halab'))

data = json.loads(query.text)

Amit Tripathi · Accepted Answer · 2017-11-28T17:50:22.523

4

Since Wikipedia doesn't have a proper 301/302 redirection what happens when you open the link is a proper 200 success response is returned and then url is changed using JS

I came up with a quick workable solution. First, remove &redirect=no from the URL

In [42]: import requests

In [43]: r = requests.get('https://en.wikipedia.org/w/index.php?title=(1S)-1-Met
    ...: hyl-2,3,4,9-tetrahydro-1H-pyrido-3,4-b-indole')

In [44]: tmp = r.content.replace('<link rel="canonical" href="', 'r@ndom}-=||').
    ...: split('r@ndom}-=||')[-1]

In [45]: idx = tmp.find('"/>')

In [46]: real_link = tmp[:idx]

In [47]: real_link
Out[47]: 'https://en.wikipedia.org/wiki/Tetrahydroharman'

The real URL value is stored in <link rel="canonical" href=" tag.

You can use above method which is good enough for your use case or you can use libraries like bs4 to parse the page and the get the link or use regex the extract the link.

edited Nov 28 '17 at 17:50

answered Nov 28 '17 at 17:45

Amit Tripathi

7,003
6
32
58

Is there no other way to do it without a get request!? since get requests are rather heavy-weight:-? – tgwtdt Nov 28 '17 at 19:26
is 'r@ndom}-=||' just something that you are sure won't already exist in text!? – tgwtdt Nov 28 '17 at 19:40
btw I think you should change .content to .text – tgwtdt Nov 28 '17 at 20:39
1

@thegod since the url comes from page content, no there is no lighter way. Yes, r@ndom}-=|| is some random string I added. It should not be in page content. You can make it more variable. – Amit Tripathi Nov 30 '17 at 05:13

Kathy Razmadze · Answer 3 · 2018-12-08T10:05:12.800

0

The answer of Amit Tripathi throws an exception. this is my answer:

res = requests.get(url)
doc = lxml.html.fromstring(res.content)
for t in doc.xpath("//link[contains(@rel, 'canonical')]"):
    new_url = str(t.attrib['href'])

from my experience, there might be a redirection to the same url. so better check (url != new_url) before using new_url.

edited Dec 08 '18 at 10:05

answered Dec 07 '18 at 11:20

Kathy Razmadze

1
2

score 0 · Answer 4 · answered Jun 24 '23 at 07:30

0

Here's a very simple solution using the wikipedia library:

>>> import wikipedia
>>> page = wikipedia.page(title="(0,1)_matrix", redirect=True)
>>> page.title
'Logical matrix'

answered Jun 24 '23 at 07:30

logi-kal

7,107
6
31
43

Python - How to get the page Wikipedia will redirect me to?

4 Answers4

Linked