Text Mining: Query search

Question

I have a dictionary:

{'Farage': [0, 5, 9, 192,233,341],
 'EU': [0, 1, 5, 6, 9, 23]}

Query1: “Farage” and “EU”
Query2: “Farage” or “EU”

I need to return the documents that contain these queries. For query1, for example, the answer should be [0,5,9]. I believe the answer should be something like that but in python:

final_list = []
while x≠Null and y≠Null
    do if docID(x)=docID(y)
       then ADD(final_list, docID(x))
          x← next(x)
          y ←next(y)
        else if docID(x) < docID(y)
          then x← next(x)
          else y ←next(y)
return final_list

Please help.

Yes! I would like to extract both results depending on the user inserting AND or OR in the query. — jroc, Mar 06 '19 at 20:27

Vasilis G. · Answer 1 · 2019-03-06T20:52:25.187

1

You could create your own function using sets, a structure that Python provides and works best for your case by speeding up the process of joining and intersecting sequences of elements:

def getResults(s, argument):
    s = list(s.values())
    if argument == 'OR':
        result = s[0]
        for elem in s[1:]:
            result = sorted(set(result).union(set(elem)))
        return result
    elif argument == 'AND':
        result = s[0]
        for elem in s[1:]:
            result = sorted(set(result).intersection(set(elem)))
        return result
    else:
        return None

inDict = {'Farage': [0, 5, 9, 192,233,341], 'EU': [0, 1, 5, 6, 9, 23]}

query1 = getResults(inDict, 'AND')
query2 = getResults(inDict, 'OR')

print(query1)
print(query2)

Results:

[0, 5, 9]
[0, 1, 5, 6, 9, 23, 192, 233, 341]

Note: You can remove the sorted function if you do not want any sorting.

edited Mar 06 '19 at 20:52

answered Mar 06 '19 at 20:24

Vasilis G.

7,556
4
19
29

So you want to join and intersect all these objects? – Vasilis G. Mar 06 '19 at 20:44
You solved my example and thanks a lot. But I forgot to mention that in my real case I have thousands of objects in the dictionary and I don't know exactly where the words I search for are located or if they exist at all. ex:{'hopeless': [0, 341, 19999],'tomorrow': [0,200,242,301,398,568,712], 'sad':[1,3,5,6,8,9,18,...], 'work': [13,20,45,129,...] ,etc...] what I am tryingt to say is that I have more then s1 and s2. How could I solve this? – jroc Mar 06 '19 at 20:47
@jroc Edited my question, you can try it now. – Vasilis G. Mar 06 '19 at 20:52

mad_ · Accepted Answer · 2019-03-06T21:09:23.493

You can create a dict of operators and throw set operations to get the final results. It assumes that queries follow strict rule of key1 operator key2 operator key3

For arbitrary number of arguments

import operator
d1={'Farage': [0, 5, 9, 192,233,341],
    'EU': [0, 1, 5, 6, 9, 23],
    'hopeless': [0, 341, 19999]}

d={'and':operator.and_,
  'or':operator.or_}

Queries= ['Farage and EU','Farage and EU or hopeless','Farage or EU']

for query in Queries:
    res=set()
    temp_arr = query.split()
    k1 = temp_arr[0]

    for value in range(1,len(temp_arr),2):
        op = temp_arr[value]
        k2 = temp_arr[value+1]
        if res:
            res = d[op](res, set(d1.get(k2, [])))
        else:
            res = d[op](set(d1.get(k1, [])), set(d1.get(k2, [])))
    print(res)

Output

set([0, 9, 5])
set([0, 192, 5, 233, 9, 19999, 341])
set([0, 192, 5, 6, 1, 233, 23, 341, 9])

score 0 · Answer 3 · answered Mar 06 '19 at 20:53

Bare in mind, use the conversion into sets:

>>> d = {'Farage': [0, 5, 9, 192, 233, 341] , 'EU': [0, 1, 5, 6, 9, 23]}
>>> d
{'EU': [0, 1, 5, 6, 9, 23], 'Farage': [0, 5, 9, 192, 233, 341]}
>>>
>>> set(d['EU']) | set(d['Farage'])
{0, 1, 192, 5, 6, 9, 233, 341, 23}
>>>
>>> set(d['EU']) & set(d['Farage'])
{0, 9, 5}
>>>
>>> set(d['EU']) ^ set(d['Farage'])
{192, 1, 23, 233, 341, 6}
>>>
>>> set(d['EU']) - set(d['Farage'])
{1, 6, 23}

Or change the format of the input if it is possible for the dictionary to be directly in the form of the set, that is:

>>> d = {'Farage': {0, 5, 9, 192, 233, 341}, 'EU': {0, 1, 5, 6, 9, 23}}
>>> d['EU'] & d['Farage']
{0, 9, 5}

Text Mining: Query search

3 Answers3