Saving time series data in MongoDB collection

[Market_Learner]

I am trying to save time series data (generated using web sockets) in MongoDB using Python.

from pymongo import MongoClient
client = MongoClient()
db=client.market_data    #market_data is my db name
kws = KiteTicker("4515kn****", "pOYgD0hvSfofC********")   #stock market data client connection

db.tick.insertMany(kws.connect());   

#tick is my mongo collection name kws.connect() generates timeseries data in below format.

DEBUG:root:Ticks: [{‘timestamp’: datetime.datetime(2020, 7, 20, 13, 25, 5), ‘last_price’: 1912.5, ‘oi_day_low’: 0, ‘volume’: 9522995, ‘sell_quantity’: 1213264, ‘last_quantity’: 31, ‘change’: 0.04184757022545141, ‘oi’: 0, ‘average_price’: 1913.44, ‘ohlc’: {‘high’: 1929.9, ‘close’: 1911.7, ‘open’: 1917.8, ‘low’: 1899.65}, ‘tradable’: True, ‘depth’: {‘sell’: [{‘price’: 1912.8, ‘orders’: 3, ‘quantity’: 62}, {‘price’: 1912.95, ‘orders’: 1, ‘quantity’: 61}, {‘price’: 1913.15, ‘orders’: 1, ‘quantity’: 6}, {‘price’: 1913.2, ‘orders’: 2, ‘quantity’: 194}, {‘price’: 1913.25, ‘orders’: 1, ‘quantity’: 50}], ‘buy’: [{‘price’: 1912.5, ‘orders’: 16, ‘quantity’: 1409}, {‘price’: 1912.45, ‘orders’: 1, ‘quantity’: 50}, {‘price’: 1912.25, ‘orders’: 1, ‘quantity’: 1}, {‘price’: 1912.2, ‘orders’: 1, ‘quantity’: 1}, {‘price’: 1912.15, ‘orders’: 1, ‘quantity’: 104}]}, ‘mode’: ‘full’, ‘last_trade_time’: datetime.datetime(2020, 7, 20, 13, 25, 5), ‘buy_quantity’: 806075, ‘oi_day_high’: 0, ‘instrument_token’: 738561}]

DEBUG:root:Ticks: [{‘timestamp’: datetime.datetime(2020, 7, 20, 13, 25, 6), ‘last_price’: 1912.5, ‘oi_day_low’: 0, ‘volume’: 9523089, ‘sell_quantity’: 1214242, ‘last_quantity’: 1, ‘change’: 0.04184757022545141, ‘oi’: 0, ‘average_price’: 1913.44, ‘ohlc’: {‘high’: 1929.9, ‘close’: 1911.7, ‘open’: 1917.8, ‘low’: 1899.65}, ‘tradable’: True, ‘depth’: {‘sell’: [{‘price’: 1912.8, ‘orders’: 3, ‘quantity’: 62}, {‘price’: 1912.95, ‘orders’: 1, ‘quantity’: 61}, {‘price’: 1913.15, ‘orders’: 1, ‘quantity’: 6}, {‘price’: 1913.2, ‘orders’: 2, ‘quantity’: 194}, {‘price’: 1913.25, ‘orders’: 1, ‘quantity’: 50}], ‘buy’: [{‘price’: 1912.5, ‘orders’: 17, ‘quantity’: 1354}, {‘price’: 1912.45, ‘orders’: 1, ‘quantity’: 50}, {‘price’: 1912.25, ‘orders’: 1, ‘quantity’: 1}, {‘price’: 1912.2, ‘orders’: 1, ‘quantity’: 1}, {‘price’: 1912.15, ‘orders’: 1, ‘quantity’: 104}]}, ‘mode’: ‘full’, ‘last_trade_time’: datetime.datetime(2020, 7, 20, 13, 25, 6), ‘buy_quantity’: 805788, ‘oi_day_high’: 0, ‘instrument_token’: 738561}]

db.tick.insertMany() method is not saving this data in my mongo collection.

Any help?

Thanks.

[Prasad_Saya]

Hello and welcome to the forum!

The insertMany method takes an array of documents as parameter to insert into a collection.

So, the kws.connect() returns what kind of data? It needs to be a JSON array like:
[ { id: 1, timestamp: 1234, price: 12.34 }, { id: 2, timestamp: 7890, price: 34.90 }, ... ] .

You can assign the kws.connect() returned value to a variable and use it with the insert method. In case the data is not in the required array field type, it needs some kind of transformation to an array of documents before using it with the insert.

[Market_Learner]

Thanks a lot @Prasad_Saya for your quick and timely input.

I figured out that my websocket data is not a strict JSON array [{},{},{}…] as you mentioned that it is needed for the insertMany method to work. I am checking with the publisher of this data to see if JSON array can be supported.

I will get back on this thread with updates.

Thanks again for your input.

[Market_Learner]

@Prasad_Saya - I could make the db save work. I was not using the right tick data variable which had proper JSON array format.

Thank you so much for highlighting the error.