Will this cause a deadlock or a bad pattern?-CodePudding

Will the following way of using a thread pool cause a deadlock? Or is such a pattern not preferred? If so, what is the alternative.

Passing pool to a function that is run in a thread, which in turn invokes a function that is run the same pool.

from concurrent.futures import ThreadPoolExecutor
from time import sleep


def bar():
    sleep(2)
    return 2


def foo(pool):
    sleep(2)
    my_list = [pool.submit(bar) for i in range(4)]
    return [i.result() for i in my_list]


pool = ThreadPoolExecutor(10)
my_list = [pool.submit(foo, pool) for i in range(2)]
for i in my_list:
    print(i.result())

CodePudding user response：

This would be a safe way to spawn a thread from within a thread that itself was initiated by ThreadPoolExecutor. This may not be necessary if ThreadPoolExecutor itself is thread-safe. The output shows how, in this case, there would be 10 concurrent threads.

from concurrent.futures import ThreadPoolExecutor
from time import sleep


BAR_THREADS = 4
FOO_THREADS = 2

def bar(_):
    print('Running bar')
    sleep(1)

def foo(_):
    print('Running foo')
    with ThreadPoolExecutor(max_workers=BAR_THREADS) as executor:
        executor.map(bar, range(BAR_THREADS))

with ThreadPoolExecutor(max_workers=FOO_THREADS) as executor:
    executor.map(foo, range(FOO_THREADS))

print('Done')

Output:

Running foo
Running foo
Running bar
Running bar
Running bar
Running bar
Running bar
Running bar
Running bar
Running bar
Done

CodePudding user response：

Will the following way of using a thread pool cause a deadlock? ... If so, what is the alternative?

One alternative would be to use a thread pool that does not have a hard limit on the number of workers. Unfortunately, the concurrent.futures.ThreadPoolExecutor class is not so sophisticated. You either would have to write your own, or else find one provided by a third party. (I'm not a big-time Python programmer, so I don't know of one off-hand.)

A naive alternative thread-pool might create a new worker any time submit() was called and all of the existing workers were busy. On the other hand, that could make it easy for you to run the program out of memory by creating too many threads. A slightly more sophisticated thread pool might also kill off a worker if too many other workers were idle at the moment when the worker completed its task.

More sophisticated strategies are possible, but you might have to think more deeply about the needs and patterns-of-use of the application before writing the code.