Using MongoDB for logging your data

Hey, I’ve been thinking about using mongo for my project to logging everything but I couldn’t decide that I should really us mongo for logging my data or not. Therefore, I want to ask here and get some experienced people advices.

Firstly, I want to log everything that happens during working. Thus, there’d be a lot of logs, there’d be hundreds of thousands log data. Logs will not be that large in the memory(few fields like date, user etc.)

  1. Should I use only one collection to log everything or separate every log by their category?
  2. Should I use expired data? (If it’s possible, I’d like to see logs which are have stayed longer than a year too.)
  3. Any other suggestion about this topic?

Hi @Duck,

To better assist you we need to understand how would your application access the data and how would you search update it if any.

We have some pattern for timesiries data, and it sounds like you be doing a similar task

https://www.mongodb.com/blog/post/building-with-patterns-the-bucket-pattern

We also have a great blog with a logging example
https://developer.mongodb.com/article/mongodb-schema-design-best-practices

Best
Pavel

1 Like

My application only write logs to mongo. I don’t get/update/delete logs right now.
I’ve look documents you attached. I really want to know which way is more effective for my project. Basically, I log everything about my project and I want to know the question I asked above.

Especially, I’d like to know how to store log for a long time(10 years.) without performance impact.

Hi Dusk,

In that case I think you can write the logs as single documents.

Possibly partition the data into weekly or monthly collections.

Drop collections once they past 10 years.

What are the volume of the events you log? How many per minute?

Please note that MongoDB Atlas has a datalake archiving features allowing you to archive collections to S3 Amazon buckets for cheap storage of large volumes.

https://docs.atlas.mongodb.com/online-archive/manage-online-archive

Consider this as well.

Best
Pavel

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.