Realm functions limit() and skip() are badly broken

Hello! To implement GraphQL pagination, there appear to be two options: offset-based or cursor-based. I decided to try offset-based pagination to keep my test client a little simpler. This thread indicates pagination has to be done with a custom resolver, so I started with a simple pagination function.

…or rather, I thought it would be simple. See the following function code:

exports = async ({limit = 10, offset = 0}) => {
  const db = context.services.get('mongodb-atlas').db('DATABASE_NAME');
  const results = await db.collection('COLLECTION_NAME')
      .find({})
      .sort({id: 1})  // Sort by ID
      .skip(offset)   // Skip `offset` results.
      .limit(limit)   // Limit to `limit` results
      .toArray();

  console.log('results returned', results.length);
  return results;
}

For a collection of size 100 with default limit 10 and offset 0:
Expected number of results: 10 (matching limit)
Actual value printed: 90 – what?

If you flip the order of operations and do the limit before the skip, the result is always 0.

This seems like it should work. The API limitations guide does not list limit or skip as unsupported options for the find command. The Realm docs don’t mention limit/skip at all.

The only docs I could find on skip/find in JS explicitly state:

The order in which you call limit and sort does not matter

This is definitely not true in Realm Functions (although I’m not sure if these docs apply, which is confusing as this is MongoDB running in Node, but the find() API is a little different with projections in Realm vs. args in Node).

Concrete suggestions:

  • If skip and limit don’t work, (1) don’t let them be called at all – returning an invalid result is worse than an error, (2) update the find unsupported operation list
  • Update the MongoDB Realm Atlas service docs to explicitly mention these aren’t supported.
  • In the tutorial for custom resolvers, make it clear the access to MongoDB is different than a NodeJS driver + link the relevant docs.
  • Provide examples for implementing basic pagination with a custom resolver if that’s the recommended solution (even if it can’t be fully implemented on the server yet)

Whew! That ends my functions adventure for the evening. :wink: Thanks for all of your work on Realm, and I hope these suggestions are helpful for the future. :smiley:

2 Likes

Hey Matthew -

Sorry that you had to run through so many hoops while implementing pagination.

The API limitations guide that you linked does show skip as an unsupported operation with find. It also doesn’t show skip as a supported operation at all. I’d love to get your feedback on how we can make this a more clear on our end and pass this onto the documentation team.

The fact that limit() does not work when preceded by a skip() is a bug we’re investigating. I’m happy to keep this thread updated when it is resolved.

In general (as pointed out in the thread you linked), we recommend using find() and limit() when implementing pagination due to the skip() limitation.

  • In the tutorial for custom resolvers, make it clear the access to MongoDB is different than a NodeJS driver + link the relevant docs.
  • Provide examples for implementing basic pagination with a custom resolver if that’s the recommended solution (even if it can’t be fully implemented on the server yet)

This is also feedback I will pass onto the documentation team.

3 Likes

Thanks for the reply. You’re absolutely right – the limitations guide does list skip for find as unsupported. That is entirely my mistake!

As another suggestion: a function returning bad results is worse to me than a function returning no results (i.e. calling skip should throw an error or not compile at all rather than returning something nondeterministic). That will make it very explicit that the operation isn’t supported [although I appreciate the docs too!].

Thanks for passing suggestions along!

Hey Matthew - just looked into this issue deeper and I realize I made some incorrect statements in the above post. skip() is not supported with find() in the Wireprotocol - this is not what you were using, therefore my links to the docs were not valid.

What you ran into was a simple bug that should be resolved as of 2 weeks ago. If you see any errors in the behavior now (I just tried your snippet with limit 5 and offset 0 and it returned 5 documents as expected) let me know

1 Like

It works for me. Seems to be resolved.