I am trying to execute some (several thousand) jobs in "parallel". The problem is each job can potentially contains a request to an external server, requires several MB memory. So just executing them async is a bad idea as this will cause the external server to overload and RAM usage will explode.

So I an trying to execute those jobs with limited concurrency. Based on my research and my ability to understand the other solutions I found, I decided to use async.parallelLimit() and tried to use it this way:

async function doSomething(jobDataList) {
    let tasks = jobDataList.map(function(jobInfo) {
        return async function() {
            await processJob(jobInfo);
        }
    });

    await async.parallelLimit(tasks, 4);
    // all jobs have to be finished here.
}

The above code does not work correctly especially at the end of the doSomething function there are still jobs running.

So how do I correctly execute my async processJob function in a way that they will be executed with limited concurrency and when doSomething ends all jobs have been processed (so I can continue with the next batch of jobs).

CodePudding user response：

I think the answer of Slebetman can resolve your question answer of Slebetman

You can use asyncq lib to run parallel limit

const asyncq = require('async-q');

function timeout(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

async function processJob(name, loop) {
    await timeout(500);
    console.log(Date.now()   " "   name   " "   loop);
}

async function masterFunction(name) {
    let list = Array.from(Array(10).keys());

    let tasks = list.map(function(i) {
        return async function() {
            return processJob(name, i);
        }
    });

    let result = await asyncq.parallelLimit(tasks, 2);
    console.log('all jobs have to be finished here.');
}

(async() => {
    await masterFunction("a");
    await masterFunction("b");
}).call();