4

It is very quick to query in robomongo, below pic show I do retrieve 500 doc back:

enter image description here

I have created index at result.type, my test code:

def get_test_data(limit):
    return collection.find({'result.type':'detail'})[:limit]

def test_one_read_multi_process():

    print('mongodb read')
    t = Timer()
    TASKS = list(get_test_data(500))
    print(t.elapsed_time, '\n')

pymongo takes 21.716s , it is too slow.

test on mongodb 3.6, pymongo latest, python 2&3

Maybe relate to Pymongo significantly slower than mongo shell?

But I want a solution .

KARTHIKEYAN.A
  • 18,210
  • 6
  • 124
  • 133
Mithril
  • 12,947
  • 18
  • 102
  • 153

2 Answers2

0

Maybe you can install pymongo with c extension, to check if you have already done:

import pymongo
pymongo.has_c()

False means not while True means yes.

To install pymongo with c extension, you need:

# Ubuntu or Debian
sudo apt-get install build-essential python-dev

# Red Hat based distributions
sudo yum install gcc python-devel

And then pip install pymongo.

There are some reference:

Sraw
  • 18,892
  • 11
  • 54
  • 87
0

So far get_test_data gets all the documents in the collection and slices the result.

You want to apply limit to the query.

def get_test_data(limit):
    return collection.find({'result.type':'detail'}, limit=limit)
Oluwafemi Sule
  • 36,144
  • 1
  • 56
  • 81
  • This is equal to slice. As I remanber, older version even not support limit option, only slice. PS: pymongo 's find query is iterable, so slice is fine. That's why I use list(xxx) to convert it. – Mithril Jan 04 '18 at 00:40
  • No, both are not equal. This answer applies to newer version but then you're using 3.6. From the generated command executed against Mongo server limit is applied to the query when limit is passed https://github.com/mongodb/mongo-python-driver/blob/3.6.0/pymongo/message.py#L173 – Oluwafemi Sule Jan 04 '18 at 07:31