Repository navigation
test(django-google-spanner): run live Spanner test apps in parallel with unique DB per app - #18983
sakthivelmanii wants to merge 3 commits into
Conversation
…ith unique DB per app
## Problem
- `run_testing_worker.py` invokes `django_test_suite_6.0.sh` against a live Cloud Spanner instance without setting `RUNNING_SPANNER_BACKEND_TESTS=1`, leaving `supports_foreign_keys = True` and `sql_create_fk` enabled during Django test database setup and causing every test app to execute synchronous `ALTER TABLE ... ADD CONSTRAINT ... FOREIGN KEY` deferred DDL operations.
- `django_test_suite_6.0.sh` and `django_test_suite_5.2.sh` reuse a single database name and execute `python3 runtests.py $DJANGO_TEST_APP` sequentially across all apps in `django_test_apps.txt` when running against a live Cloud Spanner instance.
## Solution
- Set `RUNNING_SPANNER_BACKEND_TESTS=1` in `run_testing_worker.py` and default `RUNNING_SPANNER_BACKEND_TESTS="${RUNNING_SPANNER_BACKEND_TESTS:-1}"` in `django_test_suite_6.0.sh` and `django_test_suite_5.2.sh` so foreign key DDL constraints are skipped, unsupported tests in `DatabaseFeatures.skip_tests` are marked skipped, and test column names are sanitized.
- Generate a dedicated database name and settings file per test app with a unique random ID per run (`${APP_PREFIX}_${RUN_ID}`) in `django_test_suite_6.0.sh` and `django_test_suite_5.2.sh`, and run all test apps concurrently in parallel on the same instance when running against live Cloud Spanner (`SPANNER_EMULATOR_HOST` unset).
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces parallel execution for Django test suites when running against a live Google Cloud Spanner instance. It generates unique database names and settings files per app to support concurrency. However, a critical issue was identified: if SPANNER_TEST_DB is predefined, the database name and settings file will be identical across all test apps, leading to conflicts and test failures during parallel execution. It is recommended to append the app prefix to SPANNER_TEST_DB to guarantee uniqueness while respecting Spanner's 30-character database ID limit.
| TEST_DBNAME="${SPANNER_TEST_DB:-${APP_PREFIX}_${RUN_ID}}" | ||
| TEST_DBNAME_OTHER="${TEST_DBNAME}-ot" | ||
| SETTINGS_FILE="${TEST_DBNAME}_settings" |
There was a problem hiding this comment.
If SPANNER_TEST_DB is set, TEST_DBNAME will be identical for all test apps. When running against a live Spanner instance (where SPANNER_EMULATOR_HOST is unset), all test apps run concurrently in parallel. This will cause them to use the exact same database name and overwrite the same settings file (SETTINGS_FILE="${TEST_DBNAME}_settings"), leading to severe conflicts and test failures.
To prevent this, we should ensure that TEST_DBNAME is unique per app even when SPANNER_TEST_DB is provided. For example, we can append the APP_PREFIX to SPANNER_TEST_DB. Keep in mind that Cloud Spanner database IDs have a strict limit of 30 characters, so we should be mindful of the length.
| TEST_DBNAME="${SPANNER_TEST_DB:-${APP_PREFIX}_${RUN_ID}}" | |
| TEST_DBNAME_OTHER="${TEST_DBNAME}-ot" | |
| SETTINGS_FILE="${TEST_DBNAME}_settings" | |
| if [ -n "$SPANNER_TEST_DB" ]; then | |
| TEST_DBNAME="${SPANNER_TEST_DB}_${APP_PREFIX}" | |
| else | |
| TEST_DBNAME="${APP_PREFIX}_${RUN_ID}" | |
| fi | |
| TEST_DBNAME_OTHER="${TEST_DBNAME}-ot" | |
| SETTINGS_FILE="${TEST_DBNAME}_settings" |
There was a problem hiding this comment.
Updated TEST_DBNAME to append _${APP_PREFIX} when SPANNER_TEST_DB is set so parallel workers don't collide on the same database, and removed the redundant RUNNING_SPANNER_BACKEND_TESTS export.
| TEST_DBNAME="${SPANNER_TEST_DB:-${APP_PREFIX}_${RUN_ID}}" | ||
| TEST_DBNAME_OTHER="${TEST_DBNAME}-ot" | ||
| SETTINGS_FILE="${TEST_DBNAME}_settings" |
There was a problem hiding this comment.
If SPANNER_TEST_DB is set, TEST_DBNAME will be identical for all test apps. When running against a live Spanner instance (where SPANNER_EMULATOR_HOST is unset), all test apps run concurrently in parallel. This will cause them to use the exact same database name and overwrite the same settings file (SETTINGS_FILE="${TEST_DBNAME}_settings"), leading to severe conflicts and test failures.
To prevent this, we should ensure that TEST_DBNAME is unique per app even when SPANNER_TEST_DB is provided. For example, we can append the APP_PREFIX to SPANNER_TEST_DB. Keep in mind that Cloud Spanner database IDs have a strict limit of 30 characters, so we should be mindful of the length.
| TEST_DBNAME="${SPANNER_TEST_DB:-${APP_PREFIX}_${RUN_ID}}" | |
| TEST_DBNAME_OTHER="${TEST_DBNAME}-ot" | |
| SETTINGS_FILE="${TEST_DBNAME}_settings" | |
| if [ -n "$SPANNER_TEST_DB" ]; then | |
| TEST_DBNAME="${SPANNER_TEST_DB}_${APP_PREFIX}" | |
| else | |
| TEST_DBNAME="${APP_PREFIX}_${RUN_ID}" | |
| fi | |
| TEST_DBNAME_OTHER="${TEST_DBNAME}-ot" | |
| SETTINGS_FILE="${TEST_DBNAME}_settings" |
There was a problem hiding this comment.
Updated TEST_DBNAME here as well to append _${APP_PREFIX} when SPANNER_TEST_DB is set, and removed the redundant RUNNING_SPANNER_BACKEND_TESTS export.
Problem
django_test_suite_5.2.shanddjango_test_suite_6.0.sh) against live Spanner, sharing a single database or creating separate Spanner instances per worker leads to DDL lock contention, instance quota exhaustion, or long sequential test runtimes.SPANNER_TEST_DBis provided in the environment, parallel worker invocations collide on the same database name unless each worker appends its unique app prefix.Solution
django_test_suite_5.2.shanddjango_test_suite_6.0.shto create a single shared Spanner instance per test run and run the test apps in parallel background jobs (MAX_PARALLEL=4).run_testing_worker.pyand the test suite shell scripts to generate a unique database name per app and per run (dj_<timestamp>_<random4>_<app_prefix>, or${SPANNER_TEST_DB}_${APP_PREFIX}whenSPANNER_TEST_DBis set) on the shared instance so parallel workers isolate their schema DDL and data cleanly.