Best Way to Build a Collection in Node Mongo Driver from a Directory-CodePudding

I asked a similar question yesterday to this, but the solution was very easy and did not really address my fundamental problem of not understanding flow control in asynchronous JavaScript. The short version of what I am trying to do is build a MongoDB collection from a directory of JSON files. I had it working, but I modified something and now the flow is such that the program runs to completion, and therefore closes the connection before the asynchronous insertOne() calls are executed. When the insertOne() calls are finally executed, the data is not input and I get warnings about an unhandled exception from using a closed connection.

I am new to this, so if what I am doing is not best practice (it isn't), please let me know and I am happy to change things to get it to be reliable. The relevant code basically looks like this:

fs.readDirSync(dataDir).forEach(async function(file){

    //logic to build data object from JSON file

    console.log('Inserting object '   obj['ID']);
    let result = await connection.insertOne(obj);
    console.log('Object '   result.insertedId   ' inserted.');

})

The above is wrapped in an async function that I await for. By placing a console.log() message at the end of program flow, followed by a while(true);, I have verified that all the "'Inserting object ' obj[ID]" messages are printed, but not the following "'Object ' result.insertedId ' inserted'" messages when flow reaches the end of the program. If I remove the while(true); I get all the error messages, because I am no longer blocking and obviously by that point the client is closed. In no case is the database actually built.

I understand that there are always learning curves, but it is really frustrating to not be able to do something as simple as flow control. I am just trying to do something as simple as "loop through each file, perform function on each file, close, and exit", which is remedial programming. So, what is the best way to mark a point that flow control will not pass until all attempts to insert data into the Collection are complete (either successfully or unsuccessfully, because ideally I can use a flag to mark if there were any errors)?

CodePudding user response：

I found a bit of an answer. It is a hack, but further searching on the net and the lack of responses indicate that there may be no real good way to reliably control flow with asynchronous code and callbacks. Basically, the modification is along the lines of:

fs.readDirSync(dataDir).forEach(async function(file){

    jobsOutstanding  ;

    //logic to build data object from JSON file

    console.log('Inserting object '   obj['ID']);
    let result = await connection.insertOne(obj);
    console.log('Object '   result.insertedId   ' inserted.');

    jobsOutstanding--;

})

Where jobsOutstanding is a top level variable to the module with an accessor, numJobsOutstanding().

I now wrap the close like this (with some logging to watch how the flow works):

async function closeClient(client){
    console.log("Enter closeClient()");
    if(!client || !client.topology || !client.topology.isConnected()){
        console.log("Already closed.");
    }
    else if(dataObject.numJobsOutstanding() == 0){
        await client.close();
        console.log("Closed.");
    }
    else{
        setTimeout(function(){ closeClient(client);}, 100);
    }
}

I got this one to run correctly, and the logging is interesting to visualize the asynchronous queue. I am not going to accept this answer yet to see if anyone out there knows something better.