M103 - Lab - Shard a Collection - Incorrect number of documents imported

Hi all, I can’t understand what’s happening.

I have created 2 shards with 3 replica set each one. Mongos is running. I launch the import command, I connect to mongos and asking for m103.products count I receive the right number: 516784

I proceed enabling sharding, adding an index and then adding a shard key.
I ask for shard status and I can see that balancer is running.
I wait until it ends, then I exit and try to validate the lab, but I receive the error:

Incorrect number of documents imported - make sure you import the entire dataset.

So I connect again to mongos asking again for the m103.products count and I receive 734669.

How is it possible? Why my actions result into a collection growth?

Thank you

after some time the count started returning the right number. But my question is still valid: why I see a different count number if balancing results terminated?

1 Like

So did your validate ran successfully finally?
Please check this

Lab - Shard a Collection // Script validate_lab_shard_collection fails allway

Hi, remove the first shard repl set “m103-repl”,

db.adminCommand( { removeShard: “m103-repl” } )

I did the same, and then reloaded the import, finally the number of documents were matching.

shards:
{ “_id” : “m103-repl”, “host” : “m103-repl/192.168.103.100:27001,192.168.103.100:27002,m103:27003”, “state” : 1, “draining” : true }
{ “_id” : “m103-repl-2”, “host” : “m103-repl-2/192.168.103.100:27004,192.168.103.100:27005,192.168.103.100:27006”, “state” : 1 }

1 Like

Exact same issue even i am facing today. Imported the documents twice on getting this error, count is correct. But after running the validation script, i got the error and i see a different count as mentioned by you. What was your fix for this?

Sounds like you’re running the validation script whilst chunk migration is still in progress, and I believe that there’s a bug in the way that the validation script counts documents, in that it can double count documents because it’s not actually counting the documents, instead it’s querying the metadata which holds an approximate document count, which isn’t always accurate whilst chunks are being migrated.

See @mattjavaly’s response to my query about this very issue:

The short answer is that to get the validation script to work for this lab, you need to wait until all the chunks have been migrated and are evenly balanced across the shards to the satisfaction of the mongos. Use the db.collections.count() method to monitor the number of documents that the metadata thinks are in your collection, and run the validation script once the metadata is in agreement with the actual number of documents in the collection.

2 Likes

Dropping the m103-repl shard was the solution for me as well. I spent hours trying to get this to work and this was the only thing that worked for me.

Even though I could see that it was indeed sharding across both m103-repl and m103-repl-2, I could not pass the test until I deleted m103-repl. I do question whether the validation test works when both shards are present. Based on the number of people I’ve seen with the same problem, I think there is something wrong with the validation test itself.

Here were my findings:


… in summary, just wait a couple of minutes for the chunks to truly migrate despite what sh.status() says.

In addition, in a sharded cluster, count() doesn’t return accurate results during chunk migration unless you add a filter. Or you can use itcount() instead which iterates through all the docs.

i solved the problem by restarting all the members of m103-repl replica set. but mongodb team should consider this question since it took too much time :frowning:

Hi everyone,
I had the same issue.
Did the first time. Checked documents, was ok at 516784, enabled sharding and then the count was 734669.
Dropped the db and tried again, same thing happened.
Then I found this thread, saying it is normal during migration.

I waited a while but the count stayed the same.
Then I quit the mongo shell and logged in again (to the mongos) and I noticed the count go down.
After a few minutes the count is correct at 516784 again.

TL;DR
Solution: Wait some minutes (go get a drink or something), also try a re-login to the mongo shell.

Hope this helps!
Kristof

Hi @Kristof_98641,

Thanks for sharing :slight_smile:

Yes, we are aware of this issue and we are working on it.

For any other users who are facing the same issue please do the following until we fix the issue :arrow_down: :

  1. As soon as you run this last command by filling in the value of a shard key in the Lab - Shard a Collection, run your validator script immediately in a new terminal.

    db.adminCommand( { shardCollection: “m103.products”, key: { <shard_key>: 1 } } )

  2. Or, if are already seeing the document count error then you can wait for some time so that the changes start reflecting in the metadata and then run the validation script.

Thanks,
Shubham Ranjan
Curriculum Support Engineer

4 Likes

I waited and still get the same error, but I think I did import the whole data set cause when I do db.products.count() I get 516784 which is specified in the instructions “The collection products should contain exactly 516784 documents.”

The problem is fixed

Hi @Diego_Hernandez_O_Hagan_03051,

I’m glad your issue got resolved. If you have any other query then please feel free to get back to us.

Happy Learning :slight_smile:

Thanks,
Shubham Ranjan
Curriculum Support Engineer

Imported all 516784 documents, but the count shows zero.
Closed shell and logged back in after 20 min. Same zero count.

vagrant@m103:~ reset vagrant@m103:~ mongoimport --drop /dataset/products.json --port 26000 -u “m103-admin” \

-p “m103-pass” --authenticationDatabase “admin”
–db m103 --collection products
2020-01-09T01:20:22.332+0000 connected to: localhost:26000
2020-01-09T01:20:22.333+0000 dropping: m103.products
2020-01-09T01:20:25.317+0000 [#…] m103.products 4.21MB/87.9MB (4.8%)
2020-01-09T01:20:28.317+0000 [##…] m103.products 8.63MB/87.9MB (9.8%)
2020-01-09T01:20:31.317+0000 [###…] m103.products 12.5MB/87.9MB (14.2%)
2020-01-09T01:20:34.317+0000 [####…] m103.products 16.4MB/87.9MB (18.6%)
2020-01-09T01:20:37.319+0000 [#####…] m103.products 20.5MB/87.9MB (23.3%)
2020-01-09T01:20:40.317+0000 [######…] m103.products 24.6MB/87.9MB (28.0%)
2020-01-09T01:20:43.317+0000 [#######…] m103.products 28.6MB/87.9MB (32.6%)
2020-01-09T01:20:46.319+0000 [#########…] m103.products 33.0MB/87.9MB (37.5%)
2020-01-09T01:20:49.318+0000 [##########…] m103.products 37.2MB/87.9MB (42.2%)
2020-01-09T01:20:52.317+0000 [###########…] m103.products 41.3MB/87.9MB (46.9%)
2020-01-09T01:20:55.319+0000 [############…] m103.products 45.2MB/87.9MB (51.4%)
2020-01-09T01:20:58.317+0000 [#############…] m103.products 49.2MB/87.9MB (56.0%)
2020-01-09T01:21:01.317+0000 [##############…] m103.products 53.2MB/87.9MB (60.5%)
2020-01-09T01:21:04.317+0000 [###############…] m103.products 57.2MB/87.9MB (65.1%)
2020-01-09T01:21:07.317+0000 [################…] m103.products 60.3MB/87.9MB (68.6%)
2020-01-09T01:21:10.317+0000 [#################…] m103.products 63.3MB/87.9MB (72.0%)
2020-01-09T01:21:13.317+0000 [#################…] m103.products 65.9MB/87.9MB (74.9%)
2020-01-09T01:21:16.317+0000 [##################…] m103.products 69.2MB/87.9MB (78.6%)
2020-01-09T01:21:19.324+0000 [####################…] m103.products 73.4MB/87.9MB (83.5%)
2020-01-09T01:21:22.317+0000 [#####################…] m103.products 77.6MB/87.9MB (88.3%)
2020-01-09T01:21:25.317+0000 [######################…] m103.products 81.9MB/87.9MB (93.1%)
2020-01-09T01:21:28.317+0000 [#######################.] m103.products 86.7MB/87.9MB (98.6%)
2020-01-09T01:21:29.050+0000 [########################] m103.products 87.9MB/87.9MB (100.0%)
2020-01-09T01:21:29.050+0000 imported 516784 documents
vagrant@m103:~$ mongo --port 26000 -u “m103-admin” -p “m103-pass” --authenticationDatabase “admin”
MongoDB shell version v3.6.16
connecting to: mongodb://127.0.0.1:26000/?authSource=admin&gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“7ceb43db-3db1-4acd-b4c8-32ff6ef1ab8f”) }
MongoDB server version: 3.6.16
MongoDB Enterprise mongos> db.products.count()
0

Hi @SwapanDas,

You are running the query on wrong database. You can check the current database to which you are connected by running this command.

db

You are supposed to run this query on the m103 database.

Hope it helps!

Thanks,
Shubham Ranjan
Curriculum Services Engineer

Thanks for the help. It worked.

Hi @SwapanDas,

I’m glad your issue got resolved. Please feel free to get back to us if you have any other query.

Happy Learning :slight_smile:

Thanks,
Shubham Ranjan
Curriculum Services Engineer

In case it helps anyone, I got the same issue where I was flagged for having 734669 docs instead of 516784. The process had to run for 15 minutes or so before it suddenly dropped to 570282, then 516784 immediately after that. The validation script will run OK with that count.

Jeremy

Hi @Jeremy_16271,

Thanks for sharing your experience. Please have a look at this post.

~ Shubham