The problem

I am making a multiprocessing chunk generator, but I can't get an object by its id from a child process. I am on Windows 10.

The code

import _ctypes
import classes as pycraft
import multiprocessing

def start_generaion(window):
    # We don't have to worry about infinite loops, because
    # this is going to be run in another thread, using popen.
    try:
        world = _ctypes.PyObj_FromPtr(window).model
        print(world, flush=True)
        
        while True:
            for i in world._queue:
                world.all_chunks[i].generate()
    except Exception as e:
        print(f'Error in world generation process. \n{e}', flush=True)

if __name__ == '__main__':
    world = pycraft.world()
    world = id(world)

    world_gen_process = multiprocessing.Process(target = start_generaion, args = (world,))
    world_gen_process.start()

Thanks in advance!

CodePudding user response：

The following code is passing the id of a regular object to another process and it is able to successfully recover the object from the id under Linux:

import _ctypes
import multiprocessing
import sys

def foo(id1, label):
    try:
        print(label, id1)
        v = _ctypes.PyObj_FromPtr(id1)
        print(v)
    except:
        print("Unexpected error:", sys.exc_info()[0])
    finally:
        print('finally')

if __name__ == '__main__':
    d = {'a': 1}
    id1 = id(d)
    foo(id1, 'Main process id:')
    p = multiprocessing.Process(target=foo, args=(id1, 'Subprocess id:'))
    p.start()
    p.join()

Prints:

Main process id: 140443371291200
{'a': 1}
finally
Subprocess id: 140443371291200
{'a': 1}
finally

However, when run under Windows, all that is output is:

Main process id: 1687385043136
{'a': 1}
finally
Subprocess id: 1687385043136

The subprocess abruptly terminates without executing the except or finally blocks for which I have no explanation.

The problem is that when a new process is created with the spawn method, as is the case with Windows, that process inherits nothing from the main process. Instead, a new address space is created and a Python interpreter is launched into that space and rereads the source file executing everything at global scope as its way of initializing memory in preparation for calling the worker function (foo in my example). But variables, will not have the same address in the new address space except by the greatest of coincidences.

You did not state what platform you were running under but I strongly suspect that it uses spawn to create new processes. But even under Linux although you are able to recreate the object reference, it is really just a copy of the original object (technically there are copy-on-write semantics involved, but once the reference count is increased for the object, that will result in an actual copy being made).