Atlas search with pagination slow

[Fred_Kufner]

We have 600,000 documents in our atlas collection, soon to be over 2M. Once we created the proper indexes, an atlas search with" text" or “autocomplete” returns in less than a second. We need pagination so we tried mongoose library mongoose-paginate-v2, https://www.npmjs.com/package/mongoose-paginate-v2 it takes 15 seconds!

Any suggestions?

[Marcus (Marcus Eagan)]

Out of curiosity, have you tried using the $skip aggregation stage?

[Jonathan_Gautier]

This can help you for pagination logic fast with $skip and $limit, just convert it to JS or NodeJS

def idlimit(page_size, last_id=None, query=None, collection=None):
"""Function returns `page_size` number of documents after last_id
and the new last_id.
"""

if query is None:
    query = {}

if last_id is None:
    # When it is first page
    cursor = collection.find(query).limit(page_size)
else:
    query['_id'] = {'$gt': last_id}
    cursor = collection.find(query).limit(page_size)

# Get the data
data = [x for x in cursor]

if not data:
    # No documents left
    del cursor
    return None, None

# Since documents are naturally ordered with _id, last document will
# have max id.
last_id = data[-1]['_id']

# Return data and last_id
del cursor
return data, last_id

[Fred_Kufner]

Hey jonathan, I wound up just using atlas search with limit and skip. I implemented virtual scrolling on my vue client and it works really well. Thanks for the response

[Marcus (Marcus Eagan)]

@Fred_Kufner That’s great to hear. Let us know if you have any more questions.