Docs about writing Rust tests that indirectly call Python with ::attach()?

[pawamoy]

My Rust tests fail with:

assertion ... failed: The Python interpreter is not initialized and the `auto-initialize` feature is not enabled.

Consider calling `Python::initialize()` before attempting to use Python APIs.

Is there documentation on how to write Rust tests that indirectly call Python::attach()? Should I initialize Python in each test that directly or indirectly uses it? Will the interpreter be shared between each test, or will each test use a different interpreter?

[sueun-dev]

fwiw, I tested this with pyo3 0.28.3 in a tiny Rust test crate.

Without auto-initialize, Python::try_attach(...) returned None before init, and Python::attach(...) panicked with the same “interpreter is not initialized” message. Calling Python::initialize() first made attach work, and calling initialize() twice was fine. I also set a value on sys in one attach call and read it back in a second one, so it’s the same interpreter state inside that test process.

So for tests I’d either enable auto-initialize for that test build, or use a small helper:

use pyo3::prelude::*;
use std::sync::OnceLock;

fn with_python<F, R>(f: F) -> R
where
    F: for<'py> FnOnce(Python<'py>) -> R,
{
    static INIT: OnceLock<()> = OnceLock::new();
    INIT.get_or_init(Python::initialize);
    Python::attach(f)
}

Then call with_python(...) from any test path that may indirectly touch Python.

The “shared between tests” bit is process-scoped: unit tests in the same Rust test binary share the initialized interpreter, while separate integration test binaries are separate processes, so each binary should initialize before touching Python too. I don’t see a test-specific docs page, mostly the auto-initialize feature docs plus the Python::attach / Python::initialize API docs.

[pawamoy]

Thanks a lot @sueun-dev! Yeah it would be great to have this info in a dedicated page 🙂 I'm still not sure I want my tests (which run in parallel I believe?) to share the same interpreter, so I'd be curious to learn if there's a way to use a different one for each test.

[pawamoy]

Another small obstacle I'm encountering is, how do I let Cargo/Pyo3 know what interpreter to use so that my Python module is detected and can be imported while running my Rust test suite. I figured I could try uv run cargo test, and that got rid of the ModuleNotFoundError, but now I'm getting an error like:

/media/data/dev/project/target/debug/deps/project-a46bd2021361307e: error while loading shared libraries: libpython3.13.so.1.0: cannot open shared object file: No such file or directory

[Alc-Alc]

I am not sure if this will help you, but as suggested in Configuring the Python version, is setting up PYO3_PYTHON an option for you? You can also set it up in .cargo/config.toml

You can override the Python interpreter by setting the PYO3_PYTHON environment variable, e.g. PYO3_PYTHON=python3.8, PYO3_PYTHON=/usr/bin/python3.9, or even a PyPy interpreter PYO3_PYTHON=pypy3.

[pawamoy]

Thanks @Alc-Alc! I think I tried this previously but IIRC it didn't work. I'd have to try again and report back. In the meantime I find the relevant .venv directory and insert paths into sys.path with something like:

let contains = sys_path
    .call_method1("__contains__", (&path,))?
    .extract::<bool>()?;
if !contains {
    sys_path.call_method1("insert", (0, path))?;
}

[ilgrad]

For the follow-up about libpython3.13.so.1.0: that error happens in the Linux dynamic loader, before the Rust test executable reaches Python initialization. Changing sys.path cannot fix this particular error; it affects Python imports after the process has started.

PYO3_PYTHON selects the interpreter used by PyO3's build configuration. The executable also needs to find that interpreter's shared library at runtime.

First inspect the Python selected by your project:

uv run python -c 'import sys, sysconfig; print(sys.executable); print(sysconfig.get_config_var("LIBDIR")); print(sysconfig.get_config_var("LDLIBRARY"))'

If the reported library directory contains the matching shared library, try this per-command Linux configuration:

test_python="$(uv run python -c 'import sys; print(sys.executable)')"
test_libdir="$("$test_python" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')"

PYO3_PYTHON="$test_python" \
LD_LIBRARY_PATH="$test_libdir${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
uv run cargo test

If the library is absent, fix the selected Python installation first. Pointing the loader at a different Python version's library is not a substitute.

After the executable starts, any remaining ModuleNotFoundError is a separate import-path problem.

Reference: PyO3 — dynamically embedding Python.