Strange behaviour of pymongo

Hi,

I’m looking for help for the following problem:

I develop a blockchain web application in Python 3.7, using MongoDB as a backend, so obviously I use the pymongo driver and Flask as well.

If I collect the data as elements of a response of a web request and write it to the DB (but not reading from it), the ObjectID becomes the part of the response (in-memory) and get an error 500 as the ObjectID - of course - is not a JSON serializable. I already tried to define a json decoder but it didn’t have any affect. I also tried to use a different variable and call the DB write operation from a completely separated module but the ObjectID still becomes the part of the in-memory response. So all in all, I’m suspecting that pymongo tricks me.

Do you have any recommendation, where to go from now? I really appreciate any recommendation!

Kind regards,
LaN

ps: I do apologize if my english is poor, I’m not a native speaker.

Can you post the code here please. That would help.

Here is the relevant code snippet:

block = {
            'index': len(self.chain) + 1,
            'timestamp': time(),
            'transactions': self.current_transactions,
            'proof': proof,
            'previous_hash': previous_hash or self.hash(self.chain[-1]),
        }

        # Reset the current list of transactions
        self.current_transactions = []
        self.chain.append(block)
        data = block
        dbWriter(data)
        print(block)
        # chainColl.insert_one(block)
        return block

You can see above, I tried to insert the “block” data directly from here (“chainColl” is the collection name in mongo) but then the “return block” contained the ObjectID. Then I wrote my own module (dbWriter) and used a completely different variable “data” but ObjectID also got into “block” after writing into the DB.

PyMongo automatically adds the _id field when inserting documents as described here:
https://pymongo.readthedocs.io/en/stable/faq.html#why-does-pymongo-add-an-id-field-to-all-of-my-documents

You can use PyMongo’s JSON helpers to encode these documents to MongoDB extended JSON. You’ll most likely want to use RELAXED_JSON_OPTIONS:

>>> from bson.json_util import dumps, RELAXED_JSON_OPTIONS
>>> from bson.objectid import ObjectId
>>> dumps({'_id': ObjectId(), 'list': [], 'i': 1}, json_options=RELAXED_JSON_OPTIONS)
'{"_id": {"$oid": "5e8fa5e2f8fceeb62d28b6e3"}, "list": [], "i": 1}'

To learn more about MongoDB extended JSON see:

1 Like

Hi Shane, thanks for having a look on this. If PyMongo adds the “_id” field into the document on the disk, that’s fine, I was aware. My problem is that this field also became the part of the web response, what is an in-memory object and I found it strange.

Stay safe!

What you are describing is the documented behavior of PyMongo:

>>> my_doc = {'x': 1}
>>> collection.insert_one(my_doc)
<pymongo.results.InsertOneResult object at 0x7f3fc25bd640>
>>> my_doc
{'x': 1, '_id': ObjectId('560db337fba522189f171720')}

https://pymongo.readthedocs.io/en/stable/faq.html#why-does-pymongo-add-an-id-field-to-all-of-my-documents

1 Like

Thanks for the clarification, Shane. Stay safe!