hi, i want to make a data migration tool for migrating data from 1 environment to another, these environments will be feature environments but in order to populate data into these environments there is a need for a tool that can migrate data from a mongo into this environments instance of mongo.
so i was thinking of a simple application that takes a connection string in an input field so that we can migrate from any of our current and future mongo databases, and an input field to write a simple query to specify the dataset you need as there will be a data limit as to how much data you are allowed to migrate for various reasons,
this means that i will need to be able to pass not only a connection string from a string input field, which is the easy part, but be able to write find querys in another field that i will then be able to use with the c# driver and run against the database the connection string leads to.
is there a way to do this? or does mongoDB simply not support understanding queries from strings alike how SQL understands queries from strings?
Unlike SQL, the MongoDB query language isn’t really a language, but is, instead, a set of method/function calls on the collection object. You can pass components of the arguments, like the find JSON doc and the sort JSON doc to those operations.
Typically, C# developers tend to use the builders instead.
this i know, however i was looking for a simple way of writing queries into a text input field and have it parse it as a query, i would ofcourse prefer to achieve this without programattically identiyf keywords as FIND and find some way of attaching the find function when building my function and such. but if there is no way around it then i suppose this is what i would have to do
AFAIK, MongoDB does not have a language, so there’s nothing to parse. There’s nothing to work around.
Having written this very utility that you describe, twice, there was no need to have to deal with arbitrary statements in my case. For me, the utility can be told the collections to copy and even the search criterion to be used to select documents from those collections. The operations required to migrated that data can be determined at run time and the variability of the operations is very limited. BTW, you may need to process change streams as well, depending on your use case.
Given that there is likely a time limit on the migration, like having it complete is a maintenance window, it would be unwise to have dynamic SQL-like statements even if you could because one would want to use prepared statements in the implementation in order to gain the related performance.
What benefit do you expect to gain if there was a MongoDB operation language?
well, there is no maintenance time limit, as we would populate these databases as the feature testing environments were set up. the benefit i was expecting was the ability to have a query input field, that could easily allow for writing the specific find query with whatever else you needed like order by or skip and take or what else you need, and then have it understand the query from that text input field without having to figure out how to create a factory that detects keywords from the text in order to put together the query. I know i can pass string variables to the inside of a find method in the c# driver but i would like to avoid writing a strict query that takes variables and allow the tool to construct the entire query from text.
but seeing as there is no mongo language i will need to figure out a suitable alternative, to the solution i first thought of.
Depending on your use case, you can try to utilise BsonDocument.Parse to parse a string into BsonDocument that you could pass into Collection.Find(). For example:
string query = "{ 'foo' : 'bar', 'baz':{'$gt': 7} }";
var filter = BsonDocument.Parse(query);
var result = collection.Find(filter).FirstOrDefault();
You won’t be able to construct limit and sort with Find(), but you may be able to use aggregation pipeline instead. With aggregation pipeline you could construct $match, $sort, etc
Just be extra careful exposing string(s) that your application would pass into the database. Generally you would display collections and fields within the collections for users to filter on. Essentially creating an abstraction layer for sanity check and safety.