MongoDB.live, free & fully virtual. June 9th - 10th. Register Now

Cursor iteration is very slow in pymongo

mongodb query return command cursor .to iterate the cursor and get the data its very slow takes around 90 sec for 250 records .Can someone help

using pymongo 3.9.0

Hi @Yogita_Pal,

First of all, welcome to the MongoDB Community.
Could you share your code?

All the best,

Rodrigo (a.k.a. Logwriter)

cursor= db.collection.aggregate(
    [
        {
            '$addFields':
                {

                    'yearSubstring': {'$substr': ["$__json.created_on", 0, 10]},
                }
        },
        {
            '$match': {
                '$and': [{'yearSubstring': {'$gte': '12/01/2019'}}, {'yearSubstring': {'$lte': '12/02/2019'}}]
            }
        }

    ])

#query take around 4-5 seconds
while cursor and cursor.alive:
list_element= cursor.next()
items.append(list_element)
#but this while to read each cursor data take around 100 seconds

Just call list(cursor) to create a list of the results.

1 Like

I had already done that before and tried again it also takes same takes same time .what is standard bench mark for reading 1000 records from mongodb server using pymongo 3.9 .

90 seconds for 250 records does not seem normal to me.

what is standard bench mark for reading 1000 records from mongodb server using pymongo 3.9

Assuming these are small documents (<1KB) they can all be returned in a single network roundtrip to the server and the total time should be roughly equal to the network latency.

Although in general the answer depends on a number of factors:

  • What server are you running against? MongoDB Atlas? If so, what size cluster: Free tier M0, M5, M20, etc?
  • What is the network latency from the application to the server? 10ms? 500ms?
  • What is the average size of the returned documents? ~100 bytes, ~1KB, ~1MB?
  • How long does the server take to satisfy the query? Perhaps the query can be sped up with an index?
  • Was pymongo installed with the C extensions? These speed up pymongo’s BSON encoding/decoding. You can check with: python3 -c 'import bson;print(bson.has_c())'

A final note, you can use cProfile to determine where the CPU time (not I/O time) is being spent:

1 Like