Auto-Delete Certain type of Document in Mongodb in collection

I have a collection called PeopleDocument. This collection contains three different types of files: IDCardCopy, taxCopy, PermitCopy. Users can upload any of these files. I want to autodelete IDCardCopy one year after it was uploaded. I am looking at MongoDB TTL, however I have some questions:

db.PeopleDocument.createIndex( { "lastModifiedDate": 1 }, { expireAfterSeconds: 31622400} ) 

If I create an index like the one above, I think it will delete all files in PeopleDocument after 1 year, is it possible to only delete IDCardCopy?

More detail:

This is a C# code I use to insert Document:

var collInternDocuments = _database.GetCollection<BsonDocument>("PPLDocuments");



var doc = new BsonDocument{
                             {"Documentid", strDocumentID},
                             { "FileType", typeOfDocument},                                        
                             { "FileContent", file.ContentType},
                             { "FileName", file.FileName},
                             { "UploadTime", DateTime.Now},
                             { "UploadedBy", uploadedBy},
                             { "GridFSFileID", strGridFSFileID}
                           };
    

collInternDocuments.UpdateOne( Builders<BsonDocument>.Filter.Eq("_id", internUserID), Builders<BsonDocument>.Update.AddToSet("Documents", doc));

This is how I created Index:

db.PeopleDocument.createIndex( {UploadTime:1}, 
                               {
                                  expireAfterSeconds:900,
                                  partialFilterExpression:{
                                                            FileType:{$eq:"IDCardCopy"}
                                                            }
                                })

This is the results for db.PeopleDocument.getIndexes():

 {
    "v":2,
    "key":{
           "_id":1
           },
     "name" :"_id_",
      "ns" : "people.PeopleDocument"
},
{ 

 "v":2,
    "key":{
           "UploadTime":1
           },
     "name" :"UploadTime_1",
      "ns" : "people.PeopleDocument",
      "expireAfterSeconds":900,
      "partialFilterExpression":{
               "FileType": { "$eq":"IDCardCopy"
                 }
           }
 }

This didn’t delete the file after 900 sec, could this be a date issue?

Hello @Sujinthan_Satkunaraj, welcome to the MongoDB Community forum.

I have a collection called PeopleDocument. This collection contains three different types of files: IDCardCopy, taxCopy, PermitCopy. Users can upload any of these files. I want to autodelete IDCardCopy one year after it was uploaded. I am looking at MongoDB TTL.

This will work using TTL index along with a Partial Index. The documents with FileType = “IDCardCopy” will be deleted after one year, when you specify the Partial Index filter for the “IDCardCopy” FileType. Note the TTL index field must be a BSON Date field.

db.PeopleDocument.createIndex( { “lastModifiedDate”: 1 }, { expireAfterSeconds: 31622400} )
If I create an index like the one above, I think it will delete all files in PeopleDocument after 1 year, is it possible to only delete IDCardCopy?

Yes, the lastModifiedDate must be a BSON Date field.

{ "FileType", typeOfDocument },
{ "UploadTime", DateTime.Now }

db.PeopleDocument.createIndex( 
  { UploadTime: 1 }, 
  {
    expireAfterSeconds: 900,
    partialFilterExpression: { FileType: { $eq: "IDCardCopy" } }
  }
)

The above code looks okay to me. The documents of FileType: { $eq: "IDCardCopy" } should be deleted after 900 seconds (15 mins).

Can you verify the field type of UploadTime from the mongo shell using this query?

db.PeopleDocument.aggregate([
  { $project: { UploadTimeType: { $type: "$UploadTime" } }  }
])

The output should be: "UploadTimeType" : "date"



Here is a similar example I had tried from mongo shell and this worked fine. I am using MongoDB v4.2.8.

// Collection 'test' with two documents:
{ _id:1, dt: ISODate("2020-12-07T06:54:12.562Z"), fld: 8 },
{ _id:2, dt: ISODate("2020-12-07T06:54:12.562Z"), fld: 220 }

// Create TTL Index with Partial Index
db.test.createIndex( { dt: 1 }, {
    expireAfterSeconds: 150,    // 2.5 mins
    partialFilterExpression: {
        fld: { $gt: 100 }
    }
});

db.test.getIndexes()
{
	"v" : 2,
	"key" : {
		"dt" : 1
	},
	"name" : "dt_1",
	"ns" : "test.test",
	"expireAfterSeconds" : 150,
	"partialFilterExpression" : {
			"fld" : {
				"$gt" : 100
			}
	}
}

After the index was created, the document with _id: 2 was deleted after the 2.5 mins.

I get "UploadTimeType" : "string".

So I guess UploadTime is not type: Date.
How do I insert UploadTime as type Date?

I tried the following it didn’t work:

BsonDateTime uploadDate = new BsonDateTime(DateTime.Now);
var doc = new BsonDocument{
                         {"Documentid", strDocumentID},
                         { "FileType", typeOfDocument},                                        
                         { "FileContent", file.ContentType},
                         { "FileName", file.FileName},
                         { "UploadTime", uploadDate },
                         { "UploadedBy", uploadedBy},
                         { "GridFSFileID", strGridFSFileID}
                       };

This did not work.

Can you explain what is that did not work?

Are you able to insert the document? If so, what does it look like when queried from mongo shell? Please post the result of your insert.

BsonDateTime uploadDate = new BsonDateTime(DateTime.Now); 

I thought by inserting uploadDate as a BsonDateTime it will store uploadDate as type DateTime, however that didn’t work. When I run:

db.PeopleDocument.aggregate([
{ $project: { UploadTimeType: { $type: "$UploadTime" } }  }])

it returns UploadTimeType: string.

I solved this by creating a C# console app that will delete documents created one year ago, I created a Task Scheduler that will run everyday to run this app.