I am new to storing file in mongodb. I need to store text files containing 60 fractional values. These files must be associate with a patient id, a medical parameter like body temperature. I also need to associate the file with date and time. Number of files will be in Lakhs.
What is the best approach of storing
as file in collection, taking a lot of time to store using Python. Retrieve data I will try now.
gridFS?
What if I have pdf files containing ECG image.
Kindly suggest
One option for storing files is to use gridFS. See https://docs.mongodb.com/manual/core/gridfs/ for more details.
Another option is to store the files in something like a S3 bucket.
1 Like
Thank you, I will try s3 bucket and respond if there is any problem…
You can also store small binary files (need to be smaller than 16mb) inside of the bin data type within your MongoDB documents.
For text files I’d recommend parsing into a proper MongoDB document so you can query on the data. You could also look at Atlas Data Lake for querying data in text within S3
2 Likes
Great point on storing the data in MongoDB if you’ll need to query it!
1 Like
I have not got the chance to work with all the options. Till now, I have worked with storing file in a collection and gridFS.
I need another suggestion in this respect.
If I want to store normal data fields along with 2.2MB sized files, is it OK if I keep these files in collection? May need to store multiple files (2/3) in one document.
Kindly advise.
It depends on what the files are and how you want to query them. If the files aren’t text-based, you could put them in a S3 bucket and store links to them in your documents.
If the files are text-based and you’ll want to query them, Andrew’s recommendations above are excellent. If you’re interested in learning more about Atlas Data Lake, check out MongoDB Atlas Data Federation - Available On AWS S3 | MongoDB | MongoDB. Also, https://www.mongodb.com/how-to/atlas-data-lake-setup has information on how to get setup and query the data.
1 Like
It’s totally fine to store your data fields along with the file data in the collection - in fact I’d say it was recommended - you’ll be able to more easily search on this data, for example.
Providing your total MongoDB document size doesn’t exceed 16MB, you can store the data from several files in a single MongoDB document. It depends on your circumstances (how many files multiplied by how big each file is). One option would be to store the files in buckets that would allow you to store a group of files across several documents, along with the associated data fields.
The other suggestions in this thread are all good, I’d check them all out and go with what suits your circumstances best.
1 Like
Thanks to all of you.
2 Likes