Best way to store IoT metrics without duplicates

Working with Data
1

Hi,

Am trying to find a solution to saving IoT meter data in an effective way.

I found two challanges:

  1. Avoid duplicates from meter that sends redundant updates
  2. Limit number of DB interactions for each meter update to get good performance.

One solution I found is to use a unique combined index that covers the following
MeterID, MetricID, TimeStamp

And then make a bulk insert (insertMany) with the option ordered: false so that MongoDB will insert only new metrics that is not already in the DB (based on the combined index).

When doing inserts I can by this send any update and be sure that there will be no duplicates per meter. For each meter update I only need one MongoDB request.

My question about this solutionis:
Is this a good solution?
Will I get good performance with single updateMany according to the above?
Is there any other solution that solves the same requirements?

2

Hi @Johan_Sandberg, welcome to the community!

You’re on the correct path to a good solution/implementation.

Have you looked at the Time Series Collection? In almost all IoT cases, it is the best solution. The main issue of this type of collection for your case is that you can have duplicated data and it’s impossible to use updateOne with upsert for this type of collection…

Another possible solution is to implement Bucket Pattern and maybe the Computed Pattern. For the insertion of the data and dealing with the duplicated date, I would use Bulk Write Operations with Update One and the upsert option.