How to integrate Wikidata query in python

Question

I am currently using Wikidata Query Service to run my Wikidata queries.

For example, one of my Wikidata queries looks as follows.

SELECT ?sLabel {
    SERVICE wikibase:mwapi {
        bd:serviceParam wikibase:api "EntitySearch".
        bd:serviceParam wikibase:endpoint "www.wikidata.org".
        bd:serviceParam mwapi:search "natural language processing".
        bd:serviceParam mwapi:language "en".
        ?item wikibase:apiOutputItem mwapi:item.
        ?num wikibase:apiOrdinal true.
    }
    ?s wdt:P279|wdt:P31 ?item .
    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" }
}
ORDER BY ?num
LIMIT 10

I would like to know if we can use these queries in a python program? If so, how can we integrate the queries in python?

I am happy to provide more details if needed.

it's not allowed to ask for tool, libs etc. on Stackoverflow, or better said it's considered off-topic. And using a seach engine for "Python + SPARQL" should show up `sparqlwrapper` as part of `RDFlib` as one of the first hits. — UninformedUser, May 03 '19 at 04:55
Tool requests are welcome at https://softwarerecs.stackexchange.com :-) — Nicolas Raoul, Oct 05 '20 at 08:58

score 10 · Answer 1 · answered Nov 18 '20 at 19:11

10

In case you want to do it without a SPARQL specific library:

import requests

url = 'https://query.wikidata.org/sparql'
query = '''
SELECT ?item ?itemLabel ?linkcount WHERE {
    ?item wdt:P31/wdt:P279* wd:Q35666 .
    ?item wikibase:sitelinks ?linkcount .
FILTER (?linkcount >= 1) .
SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" . }
}
GROUP BY ?item ?itemLabel ?linkcount
ORDER BY DESC(?linkcount)
'''
r = requests.get(url, params = {'format': 'json', 'query': query})
data = r.json()

answered Nov 18 '20 at 19:11

Abbe

1,765
1
22
34

can you please add code to convert to data.frame – userJT Jan 06 '21 at 19:38
see my answer below - the list of dicts can be directly imported to pandas – Wolfgang Fahl Oct 29 '21 at 17:37

score 7 · Accepted Answer · answered May 03 '19 at 00:04

7

sparqlwrapper can handle that. You can find more information here

answered May 03 '19 at 00:04

Alassane Ndiaye

4,427
1
10
19

Wolfgang Fahl · Answer 3 · 2021-10-30T08:51:26.490

As the committer of http://wiki.bitplan.com/index.php/PyLoDStorage i'd recommend this library since it wraps sparqlwrapper in a way that you immediately get proper python types. Thus you'll be able to get the data into other libaries such as pandas or work with csv, json, xml, sql or whatever your followup needs are.

I have modified and extended your query a bit

# WF 2021-10-30
# see https://stackoverflow.com/questions/55961615/how-to-integrate-wikidata-query-in-python/69771615#69771615
SELECT ?s ?sLabel ?item ?itemLabel ?sourceCode ?webSite ?stackexchangeTag  {
    SERVICE wikibase:mwapi {
        bd:serviceParam wikibase:api "EntitySearch".
        bd:serviceParam wikibase:endpoint "www.wikidata.org".
        bd:serviceParam mwapi:search "natural language processing".
        bd:serviceParam mwapi:language "en".
        ?item wikibase:apiOutputItem mwapi:item.
        ?num wikibase:apiOrdinal true.
    }
    ?s wdt:P279|wdt:P31 ?item .
    OPTIONAL { 
      ?s wdt:P1324 ?sourceCode.
    }
    OPTIONAL {    
      ?s wdt:P856 ?webSite.
    }
    OPTIONAL {    
      ?s wdt:P1482 ?stackexchangeTag.
    }
    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" }
}
ORDER BY ?itemLabel ?sLabel

try it!

and then used it as a showcase for the pyLoDStorage library:

def testStackoverflow55961615Query(self):
        '''
        see 
        https://stackoverflow.com/questions/55961615/how-to-integrate-wikidata-query-in-python
        https://stackoverflow.com/a/69771615/1497139
        '''
        endpoint="https://query.wikidata.org/sparql"
        wd=SPARQL(endpoint)
        queryString="""SELECT ?s ?sLabel ?item ?itemLabel ?sourceCode ?webSite ?stackexchangeTag  {
    SERVICE wikibase:mwapi {
        bd:serviceParam wikibase:api "EntitySearch".
        bd:serviceParam wikibase:endpoint "www.wikidata.org".
        bd:serviceParam mwapi:search "natural language processing".
        bd:serviceParam mwapi:language "en".
        ?item wikibase:apiOutputItem mwapi:item.
        ?num wikibase:apiOrdinal true.
    }
    ?s wdt:P279|wdt:P31 ?item .
    OPTIONAL { 
      ?s wdt:P1324 ?sourceCode.
    }
    OPTIONAL {    
      ?s wdt:P856 ?webSite.
    }
    OPTIONAL {    
      ?s wdt:P1482 ?stackexchangeTag.
    }
    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" }
}
ORDER BY ?itemLabel ?sLabel"""
        qlod=wd.queryAsListOfDicts(queryString,fixNone=True)
        query=Query(name="EntitySearch",query=queryString,lang='sparql')
        debug=self.debug
        for tablefmt in ["github","mediawiki","latex"]:
            lod=copy.deepcopy(qlod)
            qdoc=query.documentQueryResult(lod,tablefmt=tablefmt)
            if debug:
                print (qdoc)

And the output could be pasted here immediately as shown below. The output is also available in any format supported by the tabulate library such as mediawiki, latex and others.

EntitySearch

query

SELECT ?s ?sLabel ?item ?itemLabel ?sourceCode ?webSite ?stackexchangeTag  {
    SERVICE wikibase:mwapi {
        bd:serviceParam wikibase:api "EntitySearch".
        bd:serviceParam wikibase:endpoint "www.wikidata.org".
        bd:serviceParam mwapi:search "natural language processing".
        bd:serviceParam mwapi:language "en".
        ?item wikibase:apiOutputItem mwapi:item.
        ?num wikibase:apiOrdinal true.
    }
    ?s wdt:P279|wdt:P31 ?item .
    OPTIONAL { 
      ?s wdt:P1324 ?sourceCode.
    }
    OPTIONAL {    
      ?s wdt:P856 ?webSite.
    }
    OPTIONAL {    
      ?s wdt:P1482 ?stackexchangeTag.
    }
    SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" }
}
ORDER BY ?itemLabel ?sLabel

result

s	sLabel	item	itemLabel	stackexchangeTag	sourceCode	webSite
http://www.wikidata.org/entity/Q24841819	Q24841819	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q1898737	Morphological analysis	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q6913444	Morphological parsing	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q104415642	Wikification	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q51751772	biomedical natural language processing	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q46346005	computer-based question classification	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q1513879	natural language generation	http://www.wikidata.org/entity/Q30642	natural language processing	https://stackoverflow.com/tags/nlg
http://www.wikidata.org/entity/Q1078276	natural language understanding	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q1271424	part-of-speech tagging	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q105171570	speaker verification	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q189436	speech recognition	http://www.wikidata.org/entity/Q30642	natural language processing	https://ai.stackexchange.com/tags/speech-recognition
http://www.wikidata.org/entity/Q189436	speech recognition	http://www.wikidata.org/entity/Q30642	natural language processing	https://cs.stackexchange.com/tags/speech-recognition
http://www.wikidata.org/entity/Q189436	speech recognition	http://www.wikidata.org/entity/Q30642	natural language processing	https://dsp.stackexchange.com/tags/speech-recognition
http://www.wikidata.org/entity/Q189436	speech recognition	http://www.wikidata.org/entity/Q30642	natural language processing	https://linguistics.stackexchange.com/tags/speech-recognition
http://www.wikidata.org/entity/Q189436	speech recognition	http://www.wikidata.org/entity/Q30642	natural language processing	https://stackoverflow.com/tags/speech-recognition
http://www.wikidata.org/entity/Q189436	speech recognition	http://www.wikidata.org/entity/Q30642	natural language processing	https://unix.stackexchange.com/tags/speech-recognition
http://www.wikidata.org/entity/Q1948408	text segmentation	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q3484781	text simplification	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q2438971	tokenization	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q105330879	word segmentation	http://www.wikidata.org/entity/Q30642	natural language processing
http://www.wikidata.org/entity/Q7095836	Apache OpenNLP	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/opennlp	http://svn.apache.org/repos/asf/opennlp	http://opennlp.apache.org
http://www.wikidata.org/entity/Q7095836	Apache OpenNLP	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/opennlp	https://github.com/apache/opennlp	http://opennlp.apache.org
http://www.wikidata.org/entity/Q7095836	Apache OpenNLP	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/opennlp	https://github.com/apache/opennlp.git	http://opennlp.apache.org
http://www.wikidata.org/entity/Q7095836	Apache OpenNLP	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/opennlp	http://svn.apache.org/repos/asf/opennlp	https://opennlp.apache.org/
http://www.wikidata.org/entity/Q7095836	Apache OpenNLP	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/opennlp	https://github.com/apache/opennlp	https://opennlp.apache.org/
http://www.wikidata.org/entity/Q7095836	Apache OpenNLP	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/opennlp	https://github.com/apache/opennlp.git	https://opennlp.apache.org/
http://www.wikidata.org/entity/Q975453	General Architecture for Text Engineering	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit		https://github.com/GateNLP	http://gate.ac.uk
http://www.wikidata.org/entity/Q5533567	Gensim	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit		https://github.com/RaRe-Technologies/gensim	https://radimrehurek.com/gensim/
http://www.wikidata.org/entity/Q48688669	LingPipe	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/lingpipe		http://alias-i.com/lingpipe/
http://www.wikidata.org/entity/Q6553971	LinguaStream	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit
http://www.wikidata.org/entity/Q5396532	MALLET	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit			http://mallet.cs.umass.edu/
http://www.wikidata.org/entity/Q6906675	MontyLingua	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit			http://web.media.mit.edu/~hugo/montylingua/
http://www.wikidata.org/entity/Q3630063	Moses	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit		https://github.com/moses-smt/mosesdecoder	http://www.statmt.org/moses
http://www.wikidata.org/entity/Q1635410	Natural Language Toolkit	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit	https://stackoverflow.com/tags/nltk	https://github.com/nltk/nltk	http://nltk.org/
http://www.wikidata.org/entity/Q17071870	NiuTrans	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit			http://www.nlplab.com/NiuPlan/NiuTrans.html
http://www.wikidata.org/entity/Q47405152	Pattern	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit		https://github.com/clips/pattern	http://www.clips.ua.ac.be/pages/pattern
http://www.wikidata.org/entity/Q32998961	Stanford CoreNLP	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit			https://stanfordnlp.github.io/CoreNLP/
http://www.wikidata.org/entity/Q104840874	Stanza	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit		https://stanfordnlp.github.io/stanza/	https://stanfordnlp.github.io/stanza/
http://www.wikidata.org/entity/Q2593410	WordSmith	http://www.wikidata.org/entity/Q21129801	natural language processing toolkit			http://www.lexically.net

test testStackoverflow55961615Query, debug=False took   0.5 s
----------------------------------------------------------------------
Ran 1 test in 0.503s

OK

How to integrate Wikidata query in python

3 Answers3

EntitySearch

query

result