Store aggregation in pure JSON

I’m storing aggregations for later usage. However, I cannot get $match stages with date comparisons to work. For example, our documents have a _created_at property that is stored as a date. I was attempting to build a simple pipeline that gets documents that are newer than 7 days. I tried storing the date value in the aggregation as a standard timestamp value, like this:

[ { “$match”: { _created_at: { “$gt”: 1617311291067 } } } ]

But that did not work. How can I properly store an aggregation pipeline as pure JSON so that when I use it later (without additional processing), it will work?

I can’t store a javascript Date object - 'cause that just turns it into a date string, which also did not work, for example, this did not work:

[ { “$match”: { _created_at: { “$gt”: “2021-04-01” } } } ]

But doing this in javascript did work:

[ { “$match”: { _created_at: { “$gt”: new Date(“2021-04-01”) } } } ])

Thanks,
-Chris

Based on a reading of the mongodb c and nodejs driver code, what I asked about above is not possible. The drivers use knowledge of the data types (specifically the Date or ISODate types) to properly convert the pipeline into the binary wire format acceptable on the server side. Storing a date as string and not providing a way for the driver to know that it is a date (so it can store it as a proper BSON_DATE_TIME type) makes it impossible for the pipeline to be constructed properly internally.

Therefore, I tried another solution, which is to move the date evaluation completely to the server using the $$NOW server variable, this $match worked:

{ 
  '$match': {
    "$expr": {
      "$gt": ["$_created_at", {
        "$subtract": ["$$NOW", 604800000]
      }]
    }
  }
}

FWIW - if anyone is trying to do this we ended up experimenting for awhile and choosing another solution. The $expr operator ends up being very slow. Therefore we added our own custom operators, once of which is called $ztsfromnow. So now when we want to store a date in a JSON representation of an aggregation for later usage, we do it like this:

	{
		"$match": {
			"_created_at": {
				"$gte": {
					"$ztsfromnow": {
						"days": 7
					}
				}
			}
		}
	},

The object shape for this operator is:

interface TimeAdjustment {
  months?: number,
  weeks?: number,
  days?: number,
  hours?: number,
  minutes?: number,
  seconds?: number
}

Then when we retrieve this JSON from storage and want to execute it against one of our collections, we recursively look for our custom operators. In this case, here is the replacer function for the $ztsfromnow operator (DateTime comes from the luxon library):

  $ztsfromnow: (value: TimeAdjustment) => DateTime.now().minus(value).toJSDate(),
1 Like