Resolving a promise when wanted-CodePudding

I want to constantly sync data from a website, however I only have 300 calls/15 minutes. Thus I thought i could put all my sync requests (around 1000) into an array and then resolve just 300 of them every 15 minutes till the requests array is empty and then start again. However when I do the following:

  let requests = []
  params = 'invoice-items?invoice_id='
  let positions = null
  for (const invoice of invoices) {
    requests.push(new Promise(async (resolve, reject) => {
      positions = await getBillomatData(params   invoice.id, null, 0, null)
      await updateDatabase(positions, models.billomat.Position)
    }))
  }

  console.log(requests[0])
  await requests[0]
  console.log(requests[0])

As soon as I wait for the request at requests[0] it executes all of the requests and I go over the limit of calls.

CodePudding user response：

All async calls are executed at once because JavaScript executes ALL code as soon as each call is processed. Await awaits the result, it does not await execution.

You'll need to use a tool such as bottleneck to rate limit your requests.

Processing 300 requests of 1000 every 15 minutes will take an hour and 15 minutes to complete. This is a long time to keep the node job running doing nothing.

A basic a basic limiter might keep track of all the requests you want to make in an external file or database and then use a cron job to execute your JavaScript code every 15 minutes to process another 300 requests. If there are no additional requests, your app could just terminate. However, it will wake up and run every 15 minutes as long as the cron job keeps running.

CodePudding user response：

The most simplest approach (not optimised though) would be

make batches of 300 calls
execute one batch and wait for all of them to be resolved before proceeding to next batch

let batch = []
// imagine urls is a array of url of 900 items
urls.map(async (url)=>{
  
  batch.push(somePromiseFuctionToDoTheApiCall(url))
  if(batch.length >= 300){
    await Promise.all(batch)
    // sleep is a promisified settimeout function, ref: https://stackoverflow.com/a/56520579/3359432
    await sleep(calculateTimeToWaitBeforeProceedingToNextBatch)
    batch = []
  }

})
// there might be some leftovers at the end of batch you should process them also

if you are ok with using libraries and stop reinventing the wheel then have a look at lodash.chunk, bluebird.map, bluebird.each, bluebird.delay