NONEVM-3123: Add staging contract compatibility testing #403

jadepark-dev · 2025-12-01T14:52:09Z

This PR adds CI tests that verify plugin compatibility with staging-deployed contracts, serving as an early warning system for contract deployers and staging owners before breaking changes reach the staging environment.

Problem

When chainlink-ton/main is updated, the plugin gets redeployed to staging. However, contract deployment is manual and separate. If there's a discrepancy between the plugin version and deployed contract version, staging fails silently until someone manually tests it.

Solution

Run E2E tests against both:

Local contracts (latest changes) - existing behavior
Staging contracts (deployed version) - new compatibility check

When the staging compatibility tests fail, it indicates a breaking change is incoming.
This provides visibility into compatibility issues before they break staging, not after.

Next Steps

The current approach requires manual sync of two version pointers (.staging_contract_version in chainlink-ton and contractsVersion in chainlink-deployments).

Future improvement: Automatically resolve the staging contract version from chainlink-deployments at CI runtime, making chainlink-deployments the single source of truth. This requires additional design work around:

Cross-repo authentication (private repo access, security consideration)
Structured version file format in chainlink-deployments

patricios-space · 2025-12-05T12:30:39Z

.github/actions/ccip-e2e-setup/action.yml

+    - name: Checkout Chainlink Core repo
+      if: steps.core-cache.outputs.cache-hit != 'true'
+      uses: actions/checkout@v5
+      with:
+        repository: smartcontractkit/chainlink
+        ref: ${{ steps.read_core_ref.outputs.CORE_REF }}
+        path: chainlink
+
+    - name: Build contracts


The best way to fix this problem is to avoid checking out arbitrary, potentially untrusted commits from the Chainlink core repo in a privileged context. Ideally, workflows that deal with pull requests or other untrusted input should not check out untrusted code and run it with access to secrets or write permissions.

Instead, split the workflow into two stages:

Stage 1 (unprivileged): In response to pull requests or untrusted events, build and test the code as needed, collecting artifacts/results but not checking out arbitrary SHAs in a context with secrets.

Stage 2 (privileged): Triggered by workflow_run after Stage 1 completes successfully, download and verify the results/artifacts, and interact with secrets or make privileged changes only after verification.

For the file .github/actions/ccip-e2e-setup/action.yml, you should:

Remove or restrict the step that checks out code at an arbitrary SHA from a potentially untrusted source.

If build/test of untrusted code is absolutely necessary, ensure this workflow runs without repository secrets and with minimal permissions, and never stores results that are trusted without verification.

If privileged operations are needed, move them to a separate workflow.

In summary:

Remove the checkout step from the privileged workflow (the block starting with - name: Checkout Chainlink Core repo).

If checking out is still required, ensure only trusted refs (e.g., protected branch names, tags) may be used, or perform the checkout in an isolated and unprivileged context.

This seems to indicate this part of the workflow must be run in a separate unprivileged one?

@patricios-space Thanks for taking a look. core ref is read from a version controlled file we maintain and it's deterministic(not from untrusted PR input). Also we are not using pull_request_target, which can do more than just pull_request(like accessing repo secret) there's no privileged context to exploit.

Copilot

Pull request overview

This PR introduces automated staging contract compatibility testing to detect breaking changes before they reach the staging environment. The implementation adds a new CI workflow that validates plugin compatibility with staging-deployed contract versions, providing early visibility into compatibility issues.

Key Changes:

New staging compatibility test workflow that runs in parallel with existing integration tests
Centralized test definition file to maintain consistency across both workflows
Reusable GitHub Action for common E2E test setup, reducing code duplication

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`.github/workflows/ccip-staging-compat.yml`	New workflow for testing compatibility with staging contract versions
`.github/workflows/ccip-integration-test.yml`	Refactored to use centralized test definitions and reusable setup action
`.github/actions/ccip-e2e-setup/action.yml`	New reusable action consolidating common E2E setup steps
`.github/ccip-ton-tests.yml`	Centralized test definition file used by both workflows
`scripts/.staging_contract_version`	Version pointer for staging-deployed contracts
`scripts/e2e/lib.sh`	Modified to support external contract version configuration

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/.staging_contract_version

.github/workflows/ccip-staging-compat.yml

.github/workflows/ccip-integration-test.yml

patricios-space

I'm not a CI expert by any means, but this warning seems concerning. See https://github.com/smartcontractkit/chainlink-ton/security/code-scanning/56

krebernisak · 2025-12-10T08:37:11Z

scripts/.staging_contract_version

@@ -0,0 +1 @@
+43d7a93089fe


Let's load this from the CLD state file which should be the source of truth here, right? vs. hardcoding a magic sha

100% agree, was looking for a shortcut 😅

Exploring state update here

NONEVM-3123: store contract artifacts version in state #434

https://github.com/smartcontractkit/chainlink-deployments/pull/8633

krebernisak · 2025-12-10T08:39:53Z

.github/ccip-ton-tests.yml

@@ -0,0 +1,16 @@
+# CCIP TON E2E Test Definitions
+# This file is the single source of truth for CCIP TON integration tests.


Why is the single source truth for our tests/apps in .github CI context?

I would prefer this source of truth to be in the core project, some task manager (nix, makefile, just) to define and run common operations, so everybody can easily trigger, and then the CI uses that.

Hmm I thought this list is mostly for CI. On local development, devs usually run tests they're interested in(mostly single test). But I can see the point where CI would only pick up predefined test organization, not define by itself

krebernisak · 2025-12-10T08:47:06Z

.github/workflows/ccip-integration-test.yml

+        uses: ./.github/actions/ccip-e2e-setup
+        with:
+          contract_version: 'local'
+          build_contracts: 'true'


What's the point of this build_contracts flag you were thinking of?

I think we need it, as the above action should not build project components, but prepare a CI environment (nix, docker, cache, repo ref) in a reusable way, and then the action on top builds and prepares whatever it needs.

We define what we need in specific environments via Nix, an env like nix develop .#ccip-e2e will prepare/build/provide contracts to the e2e env, and whatever is defined in the Nix shell derivation.

What's the point of this build_contracts flag you were thinking of?

build_contracts: true → Build latest contracts from this branch

build_contracts: false → Use released/deployed version

yeah I remember there was a missing piece in e2e Nix shell, contracts flake is already available but we were building them separately. Probably we can use or override it? Let me see if I can iterate on this

krebernisak · 2025-12-10T08:50:19Z

.github/workflows/ccip-staging-compat.yml

+      - name: Load test definitions
+        id: set-matrix
+        run: |
+          MATRIX=$(yq -o=json -I=0 '.tests' .github/ccip-ton-tests.yml)


Think it would be great if immediately extend this to compat test matrix per env - it's gonna useful to run this against prod-testnet, and we can imagine we having more than one staging (sandbox?) envs.

We might have prod-testnet live, and need to patch off-chain, this will be useful to run.

We might have prod-testnet live, and need to patch off-chain, this will be useful to run.

yeah that would be great, I'll see how we can extend it targeting different environment(importing each state file seems promising)

jadepark-dev added 2 commits December 1, 2025 14:45

feat: run e2e against w/ contract version

e9ac94f

refactor: split workflows

903045f

github-advanced-security bot found potential problems Dec 1, 2025

View reviewed changes

jadepark-dev added 3 commits December 1, 2025 15:39

fix: parsing

9ea8fc7

Merge branch 'main' into jade/staging-compatibility-tests

3558ee1

chore: merge conflict

296a211

jadepark-dev changed the title ~~Add staging contract compatibility testing to CCIP E2E tests~~ NONEVM-3123: Add staging contract compatibility testing Dec 1, 2025

jadepark-dev added 4 commits December 3, 2025 11:27

fix: staging redeployment

c119c94

Merge branch 'main' into jade/staging-compatibility-tests

a8d3bc7

chore: staging redeployment

7df9940

fix: required name

e94fdc8

jadepark-dev marked this pull request as ready for review December 3, 2025 22:45

jadepark-dev requested a review from a team as a code owner December 3, 2025 22:45

Copilot AI review requested due to automatic review settings December 3, 2025 22:45

product-security-plaid-production bot requested a review from huangzhen1997 December 3, 2025 22:46

Merge branch 'main' into jade/staging-compatibility-tests

ff4dd95

Copilot AI reviewed Dec 3, 2025

View reviewed changes

scripts/.staging_contract_version Show resolved Hide resolved

.github/workflows/ccip-staging-compat.yml Show resolved Hide resolved

.github/workflows/ccip-integration-test.yml Show resolved Hide resolved

Merge branch 'main' into jade/staging-compatibility-tests

3ba83ba

patricios-space reviewed Dec 5, 2025

View reviewed changes

jadepark-dev added 3 commits December 5, 2025 18:12

feat: extra validation

4c6d825

Merge branch 'main' into jade/staging-compatibility-tests

7ba0b65

Merge branch 'main' into jade/staging-compatibility-tests

c37d82a

jadepark-dev requested review from archseer, krebernisak and nicolasgnr December 9, 2025 16:50

jadepark-dev mentioned this pull request Dec 9, 2025

chore: bumping chainlink-ton, ccip test helper clean up smartcontractkit/chainlink#20555

Merged

1 task

krebernisak reviewed Dec 10, 2025

View reviewed changes

@@ -71,15 +71,18 @@
                     path: chainlink
                     key: core-${{ steps.read_core_ref.outputs.CORE_REF }}
-                - name: Checkout Chainlink Core repo
-                  if: steps.core-cache.outputs.cache-hit != 'true'
-                  uses: actions/checkout@v5
-                  with:
-                    repository: smartcontractkit/chainlink
-                    ref: ${{ steps.read_core_ref.outputs.CORE_REF }}
-                    path: chainlink
-                    persist-credentials: false
+                # [REMOVED]: Unsafe checkout of arbitrary Chainlink Core repo commit.
+                # To safely use the Chainlink Core repo, move this step into an unprivileged workflow triggered by pull_request,
+                # or check out only trusted refs (e.g., main/master) here, never using a SHA from potentially untrusted input.
+                # For example, if only trusted branch names are allowed:
+                # - name: Safe checkout of Chainlink Core repo
+                #   if: steps.core-cache.outputs.cache-hit != 'true'
+                #   uses: actions/checkout@v5
+                #   with:
+                #     repository: smartcontractkit/chainlink
+                #     ref: main # or another trusted branch or tag
+                #     path: chainlink
+                #     persist-credentials: false
                 - name: Build contracts
                   if: inputs.build_contracts == 'true'
                   shell: bash

		@@ -0,0 +1,16 @@
		# CCIP TON E2E Test Definitions
		# This file is the single source of truth for CCIP TON integration tests.

		@@ -0,0 +1 @@
		43d7a93089fe

NONEVM-3123: Add staging contract compatibility testing #403

Are you sure you want to change the base?

NONEVM-3123: Add staging contract compatibility testing #403

Uh oh!

Conversation

jadepark-dev commented Dec 1, 2025

Problem

Solution

Next Steps

Uh oh!

Check warning

Copilot Autofix

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patricios-space left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

patricios-space left a comment •

edited

Loading