Increment over Array to use value for $addFields in aggregation

Working with Data
#1

Hello,

I have a given array of randomized numbers. For example [1,2,3…n]
Now I want to use aggregation and the $addFields function, to give each element one and only on of the values from the given array.
So document 1 has randomizedNumber = 1, document 2 has randomizedNumber = 2 and so on till document n.

I need this because I want to randomize sorting. And I need the Field, because I dont fetch all documents at once, but in small groups to let the website load smoothly. Therefore I need the constant randomized array.

Dont find any functionality. I imagine I had to loop through the given array, but doesnt find any for this in the docs.
Hope somebody can give me input for a solution.

Happy new year!

#2

Hi @Ep_Ch,

Welcome to MongoDB community.

I would like to help you out but not sure I understand the purpose or the technic you are looking for.

One of the main questions is how adding a dynamic field which is unindexed will allow the website to load smoothly, why not to fetch the documents in batches with their natural order (no order).

If the idea is the present the data randomly why not do it on the application side by randomly accessing document positions?

Once I better understand what you need I can recommend a method.

Otherwise, one way to merge the data sets us to get the fetched documents into an array and use $map or $zip to merge them into one.

Thanks
Pavel

#3

Thanks for your response, Pavel!
Excuse my bad english - its not my mothertongue.

The requirement to sort matches randomly is because fairness to the customer companies. It shall not happened, that some company-announcements for any reason is listened always behind other companies announcements, if filter-matching in the announcement-search is identitcal.
Its an easy task, when fetch all data at once, but I had to use §skip" functionality, so that I can fetch chunk-wise. Therefore its important, that the random-value I add as a field to sort in second dimension (first dimension sort is for prioritized filter-criteriy), stays at least for some time the same.

Actually I have done this dirty solution, which works, but hasnt good scalebility:

I create array and fill it with random numbers with array-length of 2000. the seed for randomization I set every hour, so the array stays constant on the one hand and chunk-wise fetching with §skip works, but also change regulary so no company is prioritized to others in the long term. Then I do $addField with randomNumber and give the value witch $arrayElementAt {randomNumberArray, elementShortID}

The elementShortId is a incremental Number every company-announce gets when insert to the DB as a document. This solution works so far, but I will get problems, when the values of the shortId increase. The length of the random array will increase more and more and its a waste of time. From a length of 2000 on you feel allready slowing down the search on the website and thats super bad user-experience.

As it is now it will work maybe for a few months, but I need a better concept, because increase of customer-announces so I can achieve the same effect but faster. I cant and shouldnt work with this up-bloated helper-array.

So Im searching for alternative concepts.

I hope I could described it better in this post here.

#4

Hi @Ep_Ch,

If you just want to show random x documents from a collection why not to use $sample which will return 200-300 random documents?

Other than that adding fields dynamically is definitely not scalable when data set grows.

What I would suggest is to use the _id or any unique based indexed field where you would populate random numbers generated during document creation.

Now when it comes to query you can randomise a seed within the existing range, example : 12211 and query the documents grater or lower than this value.

db.coll.find({_id : { $gt : 12211} })

This will give you a random set of documents everytime.

If this is not possible you should consider doing a calculated number based on a % or / math games.

Read this as well

Best
Pavel

#5

Thanks for the good input Pavel! I think I can build a fitting solution with that!

1 Like