Skip to content

echoclient.py fails on recent MacOS and Linux Python releases that default to the spawn start‑method #735

@gdahlm

Description

@gdahlm

Recent Python releases on macOS and Linux use the spawn start‑method by default for new processes.

IIRC spawn starts a brand‑new interpreter and imports the target module it relies on pickling to transport the target function.

As only module‑level objects can be pickled and run_test is defined inside the if __name__ == "__main__" block, the child process tries to unpickle and look up as an attribute of the module __mp_main__. That attribute does not exist, and the unpickler raises:

Process ForkServerProcess-2:
  File "/usr/lib/python3.14/multiprocessing/process.py", line 320, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/usr/lib/python3.14/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/concurrent/futures/process.py", line 242, in _process_worker
    call_item = call_queue.get(block=True)
  File "/usr/lib/python3.14/multiprocessing/queues.py", line 120, in get
    return _ForkingPickler.loads(res)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module '__mp_main__' has no attribute 'run_test'

I am not filing a pull request because there are at least three fixes, and which one is implemented depends on the project team's preferences.

The options I see that fit the file's stated minimal changes from upstream constraint:

  1. Move run_test to the module level.
  2. As this is I/O bound, change to the ThreadPoolExecutor
  3. Force the use of fork()
import multiprocessing
multiprocessing.set_start_method('fork', force=True)

All three options seem to benchmark close to each other on Linux/MacOS. Personally I lean towards the ThreadPoolExecutor option as that is my personal default for I/O bound concurrent operations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions