Skip to content

ci(e2e): run the 9 cypress specs that existed but were never invoked #1280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
superdav42 merged 4 commits into from
May 26, 2026
Merged

ci(e2e): run the 9 cypress specs that existed but were never invoked #1280

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d40f334
chore: upgrade planning templates
superdav42 May 20, 2026
31736aa
ci(e2e): run the 9 cypress specs that existed but were never invoked
superdav42 May 26, 2026
c50d1fc
Merge branch 'main' into chore/e2e-add-unrun-specs
superdav42 May 26, 2026
caf596d
ci(e2e): disable custom login page after wizard spec
superdav42 May 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
216 changes: 210 additions & 6 deletions .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,123 @@ jobs:
echo "=== PHP error log ==="
npx wp-env run tests-cli cat /var/www/html/wp-content/debug.log 2>&1 | tail -100 || echo "No debug.log found"

- name: Run Sanity Tests (login, mail)
id: sanity-tests
run: |
set +e
echo "=== Starting Sanity Tests ==="

# Smoke tests that don't depend on Ultimate Multisite plugin state —
# verify the WordPress test environment, admin login, and Mailpit
# service are all functional before exercising plugin features.
SANITY_TESTS=(
"tests/e2e/cypress/integration/login.spec.js"
"tests/e2e/cypress/integration/mail.spec.js"
)

TOTAL_FAILURES=0

for TEST_SPEC in "${SANITY_TESTS[@]}"; do
echo "Running: $TEST_SPEC"

npx cypress run \
--config-file cypress.config.test.js \
--spec "$TEST_SPEC" \
--browser ${{ matrix.browser }}

CYPRESS_EXIT_CODE=$?

if [ $CYPRESS_EXIT_CODE -eq 0 ]; then
echo "✅ $TEST_SPEC passed"
else
echo "❌ $TEST_SPEC failed with exit code $CYPRESS_EXIT_CODE"
TOTAL_FAILURES=$((TOTAL_FAILURES + 1))
fi
done

if [ $TOTAL_FAILURES -gt 0 ]; then
echo "❌ $TOTAL_FAILURES sanity test(s) failed"
exit 1
fi

echo "✅ All sanity tests passed!"

- name: Run Wizard Test (Setup Wizard UI)
id: wizard-test
run: |
echo "=== Starting Setup Wizard UI Test ==="

# wizard.spec.js drives the actual Setup Wizard form submission path
# (handle_save_settings -> Settings::save_settings -> Field::set_value),
# which 000-setup.spec.js bypasses by calling the installer directly.
# The wizard spec clears NETWORK_OPTION_SETUP_FINISHED itself before
# running, then re-marks setup complete at the end — safe to run after
# 000-setup. This regression-guards the addslashes/TypeError class of
# bugs that only surface when the wizard form actually POSTs.
npx cypress run \
--config-file cypress.config.test.js \
--spec "tests/e2e/cypress/integration/wizard.spec.js" \
--browser ${{ matrix.browser }}

- name: Reset custom login page (post-wizard cleanup)
id: reset-custom-login
run: |
echo "=== Disabling custom login page after wizard ==="
# The wizard's "Default Content" step creates a Login page and points
# Ultimate Multisite's custom-login feature at it, which redirects
# /wp-login.php -> /login/. The custom page has no #rememberme
# checkbox, so subsequent specs that use cy.loginByForm() fail in
# their session-setup hook. 000-setup.spec.js calls this same
# fixture defensively in its before() hook for the same reason.
PLUGIN_SLUG=$(basename "$PWD")
npx wp-env run tests-cli wp eval-file \
"/var/www/html/wp-content/plugins/$PLUGIN_SLUG/tests/e2e/cypress/fixtures/setup-disable-custom-login.php" || \
npx wp-env run tests-cli wp eval-file \
"/var/www/html/wp-content/plugins/ultimate-multisite/tests/e2e/cypress/fixtures/setup-disable-custom-login.php"

- name: Run Regression & UI Tests (After Setup)
id: regression-tests
run: |
set +e
echo "=== Starting Regression & UI Tests ==="

# High-value regression guards for previously-fixed bugs and UI
# error-handling. All are self-contained (WP-CLI fixtures or pure
# browser interaction) and need no external credentials.
REGRESSION_TESTS=(
"tests/e2e/cypress/integration/011-password-reset-subsite-domain.spec.js"
"tests/e2e/cypress/integration/030-modal-form-error-handling.spec.js"
"tests/e2e/cypress/integration/050-password-strength-enforcement.spec.js"
"tests/e2e/cypress/integration/066-sso-bootstrap-race.spec.js"
)

TOTAL_FAILURES=0

for TEST_SPEC in "${REGRESSION_TESTS[@]}"; do
echo "Running: $TEST_SPEC"

npx cypress run \
--config-file cypress.config.test.js \
--spec "$TEST_SPEC" \
--browser ${{ matrix.browser }}

CYPRESS_EXIT_CODE=$?

if [ $CYPRESS_EXIT_CODE -eq 0 ]; then
echo "✅ $TEST_SPEC passed"
else
echo "❌ $TEST_SPEC failed with exit code $CYPRESS_EXIT_CODE"
TOTAL_FAILURES=$((TOTAL_FAILURES + 1))
fi
done

if [ $TOTAL_FAILURES -gt 0 ]; then
echo "❌ $TOTAL_FAILURES regression test(s) failed"
exit 1
fi

echo "✅ All regression tests passed!"

- name: Run Checkout Tests (After Setup)
id: checkout-tests
run: |
Expand Down Expand Up @@ -232,14 +349,43 @@ jobs:

echo "✅ All checkout tests passed!"

- name: Run SSO Redirect Loop Tests (After Setup)
- name: Run SSO Tests (After Setup)
id: sso-tests
run: |
echo "=== Starting SSO Redirect Loop Tests ==="
npx cypress run \
--config-file cypress.config.test.js \
--spec "tests/e2e/cypress/integration/065-sso-redirect-loop.spec.js" \
--browser ${{ matrix.browser }}
set +e
echo "=== Starting SSO Tests ==="

SSO_TESTS=(
"tests/e2e/cypress/integration/060-sso-cross-domain.spec.js"
"tests/e2e/cypress/integration/065-sso-redirect-loop.spec.js"
)

TOTAL_FAILURES=0

for TEST_SPEC in "${SSO_TESTS[@]}"; do
echo "Running: $TEST_SPEC"

npx cypress run \
--config-file cypress.config.test.js \
--spec "$TEST_SPEC" \
--browser ${{ matrix.browser }}

CYPRESS_EXIT_CODE=$?

if [ $CYPRESS_EXIT_CODE -eq 0 ]; then
echo "✅ $TEST_SPEC passed"
else
echo "❌ $TEST_SPEC failed with exit code $CYPRESS_EXIT_CODE"
TOTAL_FAILURES=$((TOTAL_FAILURES + 1))
fi
done

if [ $TOTAL_FAILURES -gt 0 ]; then
echo "❌ $TOTAL_FAILURES SSO test(s) failed"
exit 1
fi

echo "✅ All SSO tests passed!"

- name: Run Stripe Tests (After Setup)
id: stripe-tests
Expand Down Expand Up @@ -298,6 +444,64 @@ jobs:

echo "✅ All Stripe tests passed!"

- name: Run PayPal Tests (After Setup)
id: paypal-tests
env:
PAYPAL_SANDBOX_CLIENT_ID: ${{ secrets.PAYPAL_SANDBOX_CLIENT_ID }}
PAYPAL_SANDBOX_CLIENT_SECRET: ${{ secrets.PAYPAL_SANDBOX_CLIENT_SECRET }}
run: |
if [ -z "$PAYPAL_SANDBOX_CLIENT_SECRET" ]; then
echo "⏭️ Skipping PayPal tests: PAYPAL_SANDBOX_CLIENT_SECRET not available in this run."
echo ""
echo " The repo secrets PAYPAL_SANDBOX_CLIENT_ID and PAYPAL_SANDBOX_CLIENT_SECRET"
echo " must be configured for these tests to run. GitHub Actions does NOT expose"
echo " secrets to workflows triggered by pull_request events from forked"
echo " repositories — this is a security feature, not a misconfiguration."
echo ""
echo " PayPal tests will run automatically when:"
echo " • this branch is merged to main (push event), or"
echo " • the PR is opened from a branch in the upstream Ultimate-Multisite repo"
echo " AND the PAYPAL_SANDBOX_* secrets have been configured at the repo level."
echo ""
echo " To verify locally, run: gh secret list --repo Ultimate-Multisite/ultimate-multisite"
exit 0
fi
Comment on lines +453 to +468
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE=".github/workflows/e2e.yml"
rg -n 'if \[ -z "\$PAYPAL_SANDBOX_CLIENT_ID" \] \|\| \[ -z "\$PAYPAL_SANDBOX_CLIENT_SECRET" \ ]; then|if \[ -z "\$PAYPAL_SANDBOX_CLIENT_SECRET" \ ]; then' "$FILE"

Repository: Ultimate-Multisite/ultimate-multisite

Length of output: 379

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE=".github/workflows/e2e.yml"

# Show the exact gate block around the referenced line range
nl -ba "$FILE" | sed -n '420,470p'

# Find all occurrences of PayPal secret variables
rg -n 'PAYPAL_SANDBOX_CLIENT_(ID|SECRET)' "$FILE" || true

# Look for the specific conditional check patterns (simple, no regex escapes)
rg -n 'if \[ -z "\$PAYPAL_SANDBOX_CLIENT_' "$FILE" || true

Repository: Ultimate-Multisite/ultimate-multisite

Length of output: 124

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE=".github/workflows/e2e.yml"

echo "=== lines 420-470 (with numbers) ==="
awk 'NR>=420 && NR<=470 {printf "%d\t%s\n", NR, $0}' "$FILE"

echo
echo "=== occurrences of both secret names ==="
rg -n 'PAYPAL_SANDBOX_CLIENT_(ID|SECRET)' "$FILE" || true

echo
echo "=== occurrences of the specific skip gate (secret missing) ==="
rg -n 'Skipping PayPal tests|PAYPAL_SANDBOX_CLIENT_SECRET not available|PAYPAL_SANDBOX_CLIENT_ID.*not available|if \[ -z "\$PAYPAL_SANDBOX_CLIENT_' "$FILE" || true

Repository: Ultimate-Multisite/ultimate-multisite

Length of output: 3368

Gate PayPal tests on both required secrets

.github/workflows/e2e.yml only skips when PAYPAL_SANDBOX_CLIENT_SECRET is empty; if PAYPAL_SANDBOX_CLIENT_ID is missing (empty) but the secret is present, the job will proceed and fail the tests instead of skipping.

Suggested fix 
-          if [ -z "$PAYPAL_SANDBOX_CLIENT_SECRET" ]; then
+          if [ -z "$PAYPAL_SANDBOX_CLIENT_ID" ] || [ -z "$PAYPAL_SANDBOX_CLIENT_SECRET" ]; then
             echo "⏭️  Skipping PayPal tests: PAYPAL_SANDBOX_CLIENT_SECRET not available in this run."

(Also update the skip message to reference both PAYPAL_SANDBOX_CLIENT_ID and PAYPAL_SANDBOX_CLIENT_SECRET so it matches the new condition.)

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if [ -z "$PAYPAL_SANDBOX_CLIENT_SECRET" ]; then
echo "⏭️ Skipping PayPal tests: PAYPAL_SANDBOX_CLIENT_SECRET not available in this run."
echo ""
echo " The repo secrets PAYPAL_SANDBOX_CLIENT_ID and PAYPAL_SANDBOX_CLIENT_SECRET"
echo " must be configured for these tests to run. GitHub Actions does NOT expose"
echo " secrets to workflows triggered by pull_request events from forked"
echo " repositories — this is a security feature, not a misconfiguration."
echo ""
echo " PayPal tests will run automatically when:"
echo " • this branch is merged to main (push event), or"
echo " • the PR is opened from a branch in the upstream Ultimate-Multisite repo"
echo " AND the PAYPAL_SANDBOX_* secrets have been configured at the repo level."
echo ""
echo " To verify locally, run: gh secret list --repo Ultimate-Multisite/ultimate-multisite"
exit 0
fi
if [ -z "$PAYPAL_SANDBOX_CLIENT_ID" ] || [ -z "$PAYPAL_SANDBOX_CLIENT_SECRET" ]; then
echo "⏭️ Skipping PayPal tests: PAYPAL_SANDBOX_CLIENT_SECRET not available in this run."
echo ""
echo " The repo secrets PAYPAL_SANDBOX_CLIENT_ID and PAYPAL_SANDBOX_CLIENT_SECRET"
echo " must be configured for these tests to run. GitHub Actions does NOT expose"
echo " secrets to workflows triggered by pull_request events from forked"
echo " repositories — this is a security feature, not a misconfiguration."
echo ""
echo " PayPal tests will run automatically when:"
echo " • this branch is merged to main (push event), or"
echo " • the PR is opened from a branch in the upstream Ultimate-Multisite repo"
echo " AND the PAYPAL_SANDBOX_* secrets have been configured at the repo level."
echo ""
echo " To verify locally, run: gh secret list --repo Ultimate-Multisite/ultimate-multisite"
exit 0
fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e.yml around lines 437 - 452, The conditional that gates
PayPal tests only checks PAYPAL_SANDBOX_CLIENT_SECRET; change the if-condition
to test both PAYPAL_SANDBOX_CLIENT_ID and PAYPAL_SANDBOX_CLIENT_SECRET (e.g.,
test -z "$PAYPAL_SANDBOX_CLIENT_ID" || -z "$PAYPAL_SANDBOX_CLIENT_SECRET" or a
combined -z check) so the job skips when either secret is missing, and update
the echo messages inside that block to mention both PAYPAL_SANDBOX_CLIENT_ID and
PAYPAL_SANDBOX_CLIENT_SECRET to reflect the new gating logic.


set +e
echo "=== Starting PayPal Tests ==="

PAYPAL_TESTS=(
"tests/e2e/cypress/integration/035-paypal-checkout-flow.spec.js"
)

TOTAL_FAILURES=0

for TEST_SPEC in "${PAYPAL_TESTS[@]}"; do
echo "Running: $TEST_SPEC"

npx cypress run \
--config-file cypress.config.test.js \
--spec "$TEST_SPEC" \
--browser ${{ matrix.browser }} \
--env PAYPAL_SANDBOX_CLIENT_ID="$PAYPAL_SANDBOX_CLIENT_ID",PAYPAL_SANDBOX_CLIENT_SECRET="$PAYPAL_SANDBOX_CLIENT_SECRET"

CYPRESS_EXIT_CODE=$?

if [ $CYPRESS_EXIT_CODE -eq 0 ]; then
echo "✅ $TEST_SPEC passed"
else
echo "❌ $TEST_SPEC failed with exit code $CYPRESS_EXIT_CODE"
TOTAL_FAILURES=$((TOTAL_FAILURES + 1))
fi
done

if [ $TOTAL_FAILURES -gt 0 ]; then
echo "❌ $TOTAL_FAILURES PayPal test(s) failed"
exit 1
fi

echo "✅ All PayPal tests passed!"

- name: Fix permissions for Cypress output
if: always()
run: sudo chown -R $USER:$USER tests/e2e/cypress
Expand Down
Loading