Unable to query for recent data

Working with Data Node.js Frameworks
, ,
1

I have two micro services where one write to DB and another read from DB.

The first micro service is written in C++ while the other is written in NodeJS.

When querying for data say for a IP the MongoDB return older data or no data at all (In case if the client is new) but If I query like say after 5 min it returns the recent data.

await Sessions.find({ ip: ip}).sort({ receive_time: -1 }).lean().limit(1).exec();

Every record on DB has a field called receive_time which I use to store a 64 bit timestamp .

I don’t think this has to do with concurrency at all because C++ is performant enough to write to DB faster

Worth noting, The database is in a separate server.

Ubuntu 18 LTS
C++ Mongo Driver is mongocxx 3.6.x
NodeJS Mongo Driver is Mongoose and version is 5.0

Example of Data

{
    "_id": {
        "$oid": "5f14cc1e283ea9705f1b31c2"
    },
    "userID": 1433571522,
    "ip": "135.114.236.220",
    "isNew": false,
    "receive_time": {
        "$numberLong": "1595198494512"
    },
    "event": "Click",
    "x": 93.39,
    "y": -492.00
}


{
    "_id": {
        "$oid": "5f14cc22283ea9705f1b31c3"
    },
    "userID": 1433571522,
    "ip": "135.114.236.220",
    "isNew": false,
    "receive_time": {
        "$numberLong": "1595198498608"
    },
    "event": "Press",
    "x": 91.39,
    "y": -20.00
}



{
    "_id": {
        "$oid": "5f14cc22283ea9705f1b31c4"
    },
    "userID": 1433571522,
    "ip": "135.114.236.220",
    "isNew": false,
    "receive_time": {
        "$numberLong": "1595198498652"
    },
    "event": "Type",
    "x": 11.24,
    "y": -29.00
}
2

Welcome to the MongoDB community @Ara_Threlfall!

This sounds like a timing issue in how your two different microservices interact with your MongoDB deployment.

To help understand this issue, please provide more information on your environment:

  • What specific version of MongoDB server are you using?

  • Is this MongoDB deployment a standalone, replica set, or sharded cluster?

  • What specific version of Mongoose and Node.js are you using? (Mongoose v5.0 covers a range of versions … I’m looking for the actual release version like 5.0.11).

  • How are your C++ and Node microservices interacting with the database? For example: are you writing documents in one service and reading in the other? Does the C++ microservice also query for documents but return results with the expecting timing?

It might help if you can map out a timeline of events (data written at hh:mm:ss by C++ app, Node app starts reading at hh:mm:ss).

Regards,
Stennie

3
  1. I am using MongoDB 4.4 and used this guide https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
  2. Community Server
  3. Mongoose 5.9.25
  4. I’m writing documents in one service and reading in the other.

C++ microservice also query for documents but return results with the expecting timing?

  • No

C++ app writes to two different collections and the Node APP read from two different collections

Worth noting that I didn’t use any custom index other than that default index

4

I would suggest that you take a look at https://docs.mongodb.com/manual/changeStreams/ for the reading side.

5

Change Streams doesn’t fit our use case. We’re mostly on Event Driven Arch rather Real Time.

6

Here are the timing data you requested

====================================
Two C++ Apps running in two processes (Writing)

Insert to Collection1 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection1 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection1 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection1 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:41 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 Start Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 End Time: Wed Oct 28 19:30:42 2020

Insert to Collection2 Start Time: Wed Oct 28 19:31:27 2020

Insert to Collection2 End Time: Wed Oct 28 19:31:27 2020

Insert to Collection2 Start Time: Wed Oct 28 19:31:27 2020

Insert to Collection2 End Time: Wed Oct 28 19:31:27 2020

Insert to Collection2 Start Time: Wed Oct 28 19:31:35 2020

Insert to Collection2 End Time: Wed Oct 28 19:31:35 2020

Insert to Collection2 Start Time: Wed Oct 28 19:31:35 2020

Insert to Collection2 End Time: Wed Oct 28 19:31:35 2020

Insert to Collection2 Start Time: Wed Oct 28 19:31:35 2020

Insert to Collection2 End Time: Wed Oct 28 19:31:35 2020

Insert to Collection2 Start Time: Wed Oct 28 19:31:35 2020

Insert to Collection2 End Time: Wed Oct 28 19:31:35 2020

==================
Single NodeJS App (Reading)

Start Collection2 lookup 1603913441431
Finish Collection2 lookup 1603913441453 Result is 1
Start Collection1 lookup 1603913441453
Finish Collection1 lookup 1603913441455 Result is 0

8

Does anyone can help me to resolve this?

9

In case anyone haven’t looked into timing data.

You can clearly see that C++ app finish write to DB at 19:31:35 and the NodeJS reads at 23:30:53

10

Are you able to query the recent data from the mongo shell or any other tools like Compass - after new inserts?

11

Yes, it’ shows data in Compass!

12

Hi @Ara_Threlfall,

To help clarify the problem that you’re having, could you provide the following:

  • Code snippet of the C++ insert
  • Code snippet of the Node.JS read
  • MongoDB deployment topology (standalone, replica set or sharded cluster)

It would also be useful to simplify your applications to debug the issue that you’re seeing. For example you could use just MongoDB Node.JS driver directly first, and remove Mongoose from the equation, etc.

Are you just checking this from the log file ? Are the two application log using the same timezone ? the NodeJS one is showing Unix timestamp, so that’s in UTC, but how about the C++?

Things that you should also checked are:

The issue that you’re having involved many factors and layers, as with any general debugging process, I’d encourage you to start peeling the layers one by one for deduction.

Regards,
Wan.

1 Like
13

I meant, immediately after the insert - I am assuming it is so.

14

Wan,

I’ve already wrote above that I installed MongoDB Community as it’s from your website without any further configuration.

The times are in UTC. No, I can’t share the C++ and NodeJS codebase due to my company policies. But I assure you they are correct.

15

Sometimes, correct code might still be wrong in the deployment environment. One scenario, would be that your NodeJS code reads from a delayed secondary. Since you see the data in Compass, then the data is there and the issue lies on the reading side. But not as wrong read code but as wrong deployment architecture.

16

I used this and didn’t done anything else not even created custom indexes

Why no human in this community forum can check this site?

I don’t know what you are saying by deployment method. I just used this site thats it.

17

Hi @Ara_Threlfall,

I understand your frustration, just remember that the people who has responded in this thread is trying to help you. I believe the misunderstanding here is that you have answered MongoDB Community, which is the version of MongoDB but not the deployment topology.
Now that you have clarified that you have just followed the tutorial, we can assumed that it’s a standalone deployment.

I’d recommend to create simple applications (C++ and Node.JS) to reproduce the issue that you’re seeing. You could then share the code of these simple applications so that others could help you better.

Regards,
Wan

18

Steve,

Here is the code of NodeJS

const mongoose = require('mongoose');

const Schema = mongoose.Schema;

mongoose.connect('mongodb://WHATEVERIP:27017/DBNAME?authSource=DBNAME', {

    user: "admin",

    pass: "passs",

    useNewUrlParser: true,

    useUnifiedTopology: true

});

var MarketingSchema = new Schema({}, { strict: false });

var MarketingModel = mongoose.model('Marketing', MarketingSchema, 'Marketing');

// r_time is 64 bit timestamp field

var DataRecord = await MarketingModel.find({ ip: "PUT IP OF CLIENT HERE" }).sort({ r_time: -1 }).lean().limit(1).exec();

if (!DataRecord.length) {

    console.log("NOT FOUND");

}

Here is the C++ code,

#include <iostream>

#include <string>

#include <nlohmann/json.hpp>

#include <bsoncxx/builder/stream/document.hpp>

#include <bsoncxx/json.hpp>

#include <mongocxx/client.hpp>

#include <mongocxx/stdx.hpp>

#include <mongocxx/uri.hpp>

#include <mongocxx/instance.hpp>

// Connect to Database

mongocxx::instance instance{};

mongocxx::uri uri("mongodb://user:pass@YOURIPGUESHERE:27017/?authSource=DBNAMEGOESHERE");

mongocxx::client client(uri);

mongocxx::database database = client["DBNAMEGOESHERE"];

long long getUnixTimeInMS()

{

    long long milliseconds_since_epoch = std::chrono::system_clock::now().time_since_epoch() / std::chrono::milliseconds(1);

    return milliseconds_since_epoch;

}

int main() {

    long long rtime = getUnixTimeInMS();

    // Get Marketing Table from Database

    mongocxx::collection table = ::database["Marketing"];

    nlohmann::json body;

    body["r_time"] = rtime;

    body["ip"] = "127.0.0.1";

    std::string jsonString = body.dump();

    auto document = bsoncxx::from_json(jsonString);

    table.insert_one(document.view());

    return 0;

}
19

There is no field named r_time in your sample data yet you use that field as a sort field. With the limit 1, you will end up with a random document for the given IP. Most likely in the natural order of _id, so most likely the first document inserted for the given IP which is most likely not the one you want. So it looks like a code issue.

20

That sample data was synthetic. I actually have a field called r_time

21

I don’t see this issue with MySQL but porting to MySQL is complicated for us now