As the number of possible paths grows exponentially with increasing residue count, the shortest-path computation becomes a major performance bottleneck. To address this, parallel processing was introduced for evaluating residue pairs.
However, the current implementation passes a bound method (self.calc_path_weight) to the worker pool. This causes the entire object—including the underlying NetworkX graph—to be serialized and transferred to workers repeatedly. For large graphs, this introduces significant overhead and can negate the benefits of parallelization.
A more efficient approach would be to initialize each worker once with a shared copy of the graph (e.g., via a pool initializer), so that the graph is loaded into memory a single time per worker rather than per task. This could substantially reduce inter-process communication overhead and improve performance, especially for large systems where millions of shortest-path evaluations are performed.
As the number of possible paths grows exponentially with increasing residue count, the shortest-path computation becomes a major performance bottleneck. To address this, parallel processing was introduced for evaluating residue pairs.
However, the current implementation passes a bound method (self.calc_path_weight) to the worker pool. This causes the entire object—including the underlying NetworkX graph—to be serialized and transferred to workers repeatedly. For large graphs, this introduces significant overhead and can negate the benefits of parallelization.
A more efficient approach would be to initialize each worker once with a shared copy of the graph (e.g., via a pool initializer), so that the graph is loaded into memory a single time per worker rather than per task. This could substantially reduce inter-process communication overhead and improve performance, especially for large systems where millions of shortest-path evaluations are performed.