How to parallelize code where task is dependent upon value from previous task?-CodePudding

I have some code something like the following. I would like to go ahead and start the next work item as soon as I have the nextWorkItemNumber. How could I modify this code to accomplish that?

I've never had the need to multithread something before where the next task in the loop is dependent upon a value obtained half way through the previous task.

public void DoWork()
{
    int nextWorkItem = 1;

    while(nextWorkItem > 0)
    {
       nextWorkItem = DoWork(nextWorkItem);
    }
}

public int DoWorkItem(int workItemNumber)
{
    // make service call for data
    // var nextWorkItemNumber = someNewValueBasedOnStuffAbove;
    // save results to database

    // return nextWorkItemNumber;
}

For simplicity it might help to think of the service call as a random number generator that takes a second to run. The actual scenario is more like I'm retrieving 100 records at a time from Hubspot, and I have to provide the starting point in the list from which to begin, however the list doesn't go up by ones and the only way I can give the starting point for the next request is to look at the id of the last record on the current request.

CodePudding user response：

How about this sample:

var manualResetEvent = new ManualResetEvent(false);
var nextWorkItemNumber = 0;  // not ready yet

Task.Run(() =>
{
   // this will block inside the Task, until signaled
   manualResetEvent.WaitOne();

   // nextWorkItemNumber is ready now
   // do some processing here
});

Task.Run(() =>
{
    // do some work to calculate nextWorkItemNumber 
    nextWorkItemNumber = 42; // because it's the answer to everything
    
    // signal that nextWorkItemNumber is ready
    manualResetEvent.Set();

    // keep doing more work if needed
});

CodePudding user response：

Have a look at this. I have to run now so I will add some explanation later.

void Main()
{
    DoWork();
}

public void DoWork()
{
    var saves = new List<Task<(int workItemNumber, bool success)>>();
    int nextWorkItem = 1;
    while (nextWorkItem > 0)
    {
        var result = DoWork(nextWorkItem);
        saves.Add(DoSaveAsync(result.current, result.payload));
        nextWorkItem = result.next;
    }
    Task.WaitAll(saves.ToArray());
}

private Random _random = new Random();
public (int current, int next, object payload) DoWork(int workItemNumber)
{
    var next = workItemNumber   _random.Next(5)   1;
    if (next > 20)
    {
        next = -1;
    }
    return (workItemNumber, next, new object());
}

public async Task<(int workItemNumber, bool success)> DoSaveAsync(int workItemNumber, object payload)
{
    await Task.Delay(TimeSpan.FromSeconds(1.0));
    return (workItemNumber, true);
}

For now, you'll note each save takes a second, but the overall time to run is just over one second.