Migration - Integration test fails

Data is successfully migrated, but the integration test fails… “Migration: It does not appear that you correctly converted the type”
Help please…

Please try to complete all //TODO in Migrator
*Suggestion
Bson dateStringFilter = Filters.type(“lastupdated”, “string”);
String datePattern = “yyyy-MM-dd HH:mm:ss”;
Bson ratingStringFilter = Filters.not(Filters.type(“imdb.rating”, “number”));

Hi Oudam

I’ve done all that and still get the Migration error message…

So, what is the problem you got?
Could you share a screen shot?

After running the migration, the Integration test still fails…
Even after applying your suggestion…

Bson dateStringFilter = Filters.type(“lastupdated”, “string”);
String datePattern = “yyyy-MM-dd HH:mm:ss”;
Bson ratingStringFilter = Filters.not(Filters.type(“imdb.rating”, “number”));

I think you should follow the problem description ->

  1. Update all //TODO in Migrator then
  2. Run MigrationTest to make sure everything is pass and then
  3. Run the application and then check you status

Note: if 2) have problem you need to check you Migrator methods that you applied your code again. The easy way is you need to debug your process one by one, you will see which step has problem.

Hi @Brent_07328

I would agree with @Oudam_28098’s very helpful suggestions, beyond those it might be worth re-importing your data and trying the entire migration process again.

Hope this helps!
Eoin

1 Like

Exception in thread “main” java.lang.IllegalArgumentException: Invalid BSON document for an update
at com.mongodb.operation.BulkWriteBatch$WriteRequestEncoder.encode(BulkWriteBatch.java:396)
at com.mongodb.operation.BulkWriteBatch$WriteRequestEncoder.encode(BulkWriteBatch.java:377)

I am getting this error when i tried to run migrator.java. here is my code

BulkWriteOptions bulkWriteOptions = new BulkWriteOptions();
bulkWriteOptions.ordered(false);
bulkWriteOptions.bypassDocumentValidation(true);

BulkWriteResult bulkResult = movies.bulkWrite(bulkWrites, bulkWriteOptions); // This is erroring out.

Can some one help me

Hi @zubairsmd,

Try removing the bulkWriteOptions.bypassDocumentValidation(true); line from your solution. See if that helps resolve the issue.

Let me know if you are still having an issue after that.

Best,

Yulia

Thank you for your response. Still i am getting the same error even after commenting //bulkWriteOptions.bypassDocumentValidation(true);

Hi @zubairsmd,

The issue might be in an earlier line of your other implementation. Given the error it looks like the Document is not in correct BSON format.
Could you share more info about your implementation?

Hello,
I faced with another kind of issue: migration is completed successfully (Updated 47 707 documents), in the next iteration the message “Nothing to update!” appears. All tests are passed. However during the checking status page I do see the error message “Migration: It does not appear that you correctly converted the type” and some logs:

2019-04-23 02:38:22.302 WARN 5248 — [nio-5000-exec-8] mflix.api.daos.MovieDocumentMapper : Unable to map document null to Movie object: null
2019-04-23 02:38:22.302 WARN 5248 — [nio-5000-exec-8] mflix.api.daos.MovieDocumentMapper : Skipping document

It happening with _id “573a1390f29313caabcd4132”, however the entity looks fine after migration:
image
I’ve tried to re-import data and rerun migration process - error is still here.
Does anyone faced with similar issue and found the reason?

Regards

i got this fixed, there was an issue with Iteration.

Hi @anton_rusakou,

Are you still experiencing this issue?

Hello @yulia_genkina,
Task is already locked, but I’d like to understand the reasons of such behavior. Any ideas from your side?

Thanks,
Anton

Hi anton_rusakou,

We need to see the error logs you are getting while running the integration test for Migration ticket.

Usually, we have to run migration only once. Once the type is converted, next time the expected behavior is “Nothing to Update”, which is exactly you are getting.

Kanika

This is probably because your Spring Boot application was shut down. Migrator is just a one-time-run script that does it’s job. To be able to run Integration Tests from the Frontend application your Spring-boor Backend application should be up and running.

Hi,

Try In MovieDao.getMovie find only by ID.
Or if you used unwind in MovieDao.getMovie don’t forget to set unwindOptions.preserveNullAndEmptyArrays true.

UnwindOptions unwindOptions = new UnwindOptions();
unwindOptions.preserveNullAndEmptyArrays(true);
Bson unwind = Aggregates.unwind("$comments", unwindOptions);

For this ticket MovieDao.getMovie have to be able to return ObjectId(‘573a1390f29313caabcd4132’)} (used in the status page).