MongoDB batch find performance

Working with Data
, ,
1

hi, I’m new on this forum, so i have same fetch performance problem

	topt := options.Find()
	topt.SetBatchSize(300_000)
	topt.SetLimit(300_000)
  • Deployed on Compute Engine (app & db running on difference instance)

Code :

ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

cur, err := database.Collection("chat").Find(ctx, filter, topt)
if err != nil {
    log.Println(err)
    fmt.Fprint(w, `{"success": false}`)
    return
}
defer cur.Close(ctx)

ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

var postResult []post
    err = cur.All(ctx, &postResult)
if err != nil {
    fmt.Fprint(w, `{"success": false, "message": "All failing"}`)
    return
}

returned chat length are ~36k documents and it tooks ~9sec, i need to make it faster, i really frustrated :(, any suggestions would be greatly appreciated :blush:

2

Hi @dodo_nosk,

The used index is scanning a range of dates and all of its text entires which is expected as indexes do no support a unanchored regular expressions as well as case insensitive search.

What you should consider is a text index where field text is indexed compound index

 { text : "text", timestamp : 1}

And use the $text operator with case insensitive search.

MONGODB ATLAS SEARCH

Atlas Search makes it easy to build fast, relevance-based search capabilities on top of your MongoDB data. Try it today on MongoDB Atlas, our fully managed database as a service.

Best
Pavel

3

idk, but FTS is slower than indexed-regexp on my mongo

4

Hi @dodo_nosk,

It is possible if the text index cause more work then eventually scanning the keys.

The best solution we have for regex searches is with using Atlas and its Atlas search machnics. We also have case insensitive indexes if you are looking for a specific value for case insensitive criteria .

Best regards,
Pavel