Updating Circle CI Pipeline

[manmohan-shaw-okta]

CircleCI Pipeline Configuration Guide

Overview

This CircleCI pipeline ensures secure and automated building, testing, and publishing of the Okta Python SDK.

Pipeline Structure

1. Security Scanning Jobs (Run First in Parallel)

Snyk Scan (snyk-scan)

• Runs first before build/test
• Scans for security vulnerabilities in dependencies
• Runs on every commit (including non-main branches)
• Uses the static-analysis context

Reversing Labs (reversing-labs)

• Runs first before build/test (in parallel with Snyk)
• Malware scanning using Reversing Labs scanner
• Runs on every commit
• Uses the okta-dcp context

2. Build Job (build)

• Only runs after security scans pass
• Runs on Python 3.10
• Installs dependencies from requirements.txt
• Builds distribution packages (sdist and wheel)
• Persists workspace for downstream jobs
• Does NOT run tests (separated into dedicated test jobs)

3. Unit Tests Job (unit_tests)

• Only runs after build succeeds
• Runs unit tests from tests/unit directory
• Fast-running tests that validate individual components
• Generates code coverage reports
• Stores test results and coverage artifacts

4. Integration Tests Job (integration_tests)

• Only runs after unit tests pass
• Runs integration tests from tests/integration directory
• Tests interactions between components and external systems
• Generates code coverage reports
• Stores test results and coverage artifacts

5. Publishing Job (publish_to_pypi)

• Only runs on master branch or version tags (v*)
• Only runs after all tests succeed
• Verifies distribution packages with twine check
• Publishes to PyPI using token authentication

Workflows

non-prod (Runs on PRs and non-master branches - internal contributors)

• Step 1: Snyk scan and Reversing Labs run in parallel
• Step 2: Build only if both security scans pass
• Step 3: Unit tests only if build succeeds
• Step 4: Integration tests only if unit tests pass
• Does NOT publish
• Full security scanning with context access

contributors (Runs on PRs from forked repositories - external contributors)

• Step 1: Build runs immediately (no security scans)
• Step 2: Unit tests only if build succeeds
• Step 3: Integration tests only if unit tests pass
• Does NOT publish
• Security scans are skipped (no context access for forked PRs)
• Maintainers should review code before merging

prod (Runs only on master branch)

• Step 1: Snyk scan and Reversing Labs run in parallel
• Step 2: Build only if both security scans pass
• Step 3: Unit tests only if build succeeds
• Step 4: Integration tests only if unit tests pass
• Step 5: Publish to PyPI only if all tests succeed
• Triggered by:
  • Commits to master branch
  • Version tags matching v* pattern

Required CircleCI Project Settings

Forked Pull Request Settings

To properly handle forked PRs, configure these settings in CircleCI:

1. Go to Project Settings → Advanced Settings
2. Configure the following:
   • ✅ Enable "Only build pull requests" - Only builds PRs, not random branches
   • ✅ Enable "Build forked pull requests" - Allows community contributions
   • ❌ Disable "Pass secrets to builds from forked pull requests" - Critical for security!
   • Optional: Enable "Require approval for fork pull request builds" (if available)

This configuration ensures:

• Forked PRs are built, but without access to secrets/contexts
• Security scans that require contexts will be skipped on forked PRs
• Internal team can manually trigger builds with secrets if needed
• Prevents malicious PRs from accessing your PyPI tokens or AWS credentials

Required CircleCI Contexts

You need to configure the following contexts in CircleCI:

1. static-analysis

• Used for Snyk scanning
• Should contain Snyk authentication tokens

2. okta-dcp

• Used for Reversing Labs malware scanning
• Should contain:
  • RESOURCE_TOKEN - Token for downloading RL scanner
  • AWS_ARN - AWS role ARN for authentication

3. pypi-publish

• Used for publishing to PyPI
• Should contain:
  • PYPI_TOKEN - PyPI API token (use __token__ as username)

Setting Up PyPI Token

1. Go to https://pypi.org/manage/account/token/
2. Create a new API token with appropriate scope (ideally scoped to the okta package)
3. Add it to CircleCI:
   • Go to Project Settings → Contexts
   • Create or edit the pypi-publish context
   • Add environment variable: PYPI_TOKEN = <your-token>

Security Features

1. Security-First Approach:
   • Security scans (Snyk + Reversing Labs) run before building/testing code
   • Build and test only proceed if security scans pass
   • Parallel execution of security scans for efficiency
2. Forked PR Protection via CircleCI Settings:
   • Go to Project Settings → Advanced
   • Enable "Only build pull requests" - This prevents arbitrary commits from forks from running
   • Enable "Build forked pull requests" - Allows PRs from forks after review
   • Disable "Pass secrets to builds from forked pull requests" - Prevents secrets exposure
   • This means forked PRs won't have access to contexts/secrets automatically
3. Publishing restricted to master: The publish job only runs on the master branch
4. Publishing requires successful build: Publish only runs if build and tests succeed
5. Token-based authentication: Uses secure token authentication for PyPI
6. Context protection: Security contexts (okta-dcp, static-analysis) should have restricted access

Testing the Pipeline

On Forked Repository PRs

When an external contributor submits a PR from a forked repository:

# External contributor forks and creates PR

Expected:

1. Build and test jobs run (if "Build forked pull requests" is enabled)
2. Security scans are skipped (no access to contexts)
3. Maintainer reviews the code changes
4. If approved, maintainer can manually re-run workflows with secrets enabled
5. No publishing occurs

To manually approve and run with secrets:

• Go to CircleCI workflow for the forked PR
• Click "Rerun workflow from failed"
• Enable "Rerun with SSH" or use API to run with secrets (requires admin)

On Feature Branches (Internal Repository)

git checkout -b feature/my-feature
# Make changes
git commit -m "Add new feature"
git push origin feature/my-feature

Expected: Build, test, and security scans run automatically. No publishing.

On Master Branch

git checkout master
git merge feature/my-feature
git push origin master

Expected: Build, test, security scans run, and then publishes to PyPI.

Using Version Tags

git tag v3.0.1
git push origin v3.0.1

Expected: Triggers the publish workflow with full pipeline.

Troubleshooting

How to Find Coverage Reports

Coverage reports are generated for both unit and integration tests and stored as CircleCI artifacts.

To access coverage reports:

1. Go to CircleCI: Navigate to your workflow run
2. Click on the test job: Either unit_tests or integration_tests
3. Click the "Artifacts" tab
4. Find coverage files:
   • Unit Test Coverage: coverage-unit/ folder
   • Integration Test Coverage: coverage-integration/ folder

Coverage report formats available:

• coverage.xml - Machine-readable XML format (for CI/CD tools)
• html/index.html - Human-readable HTML report with detailed line-by-line coverage
• Terminal output - Coverage summary displayed in the job logs

To view the HTML coverage report:

1. Click on coverage-unit/html/index.html or coverage-integration/html/index.html
2. The interactive HTML report shows:
   • Overall coverage percentage
   • Per-file coverage breakdown
   • Line-by-line highlighting of covered/uncovered code
   • Missing lines details

Coverage Metrics Explained:

• Statements: Percentage of code statements executed
• Missing: Number of lines not covered by tests
• Branch: Conditional branches (if/else) coverage
• Partial: Partially covered branches

Example Coverage Summary in Logs:

Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
okta/__init__.py            12      0   100%
okta/client.py             156     23    85%   45-67, 89
okta/models/user.py         89      5    94%   23, 45-48
------------------------------------------------------
TOTAL                      257     28    89%

Forked PRs Not Building

• Check Project Settings → Advanced → "Build forked pull requests" is enabled
• Verify "Only build pull requests" is enabled
• Ensure the PR is actually a pull request (not just a branch push)

Security Scans Failing on Forked PRs

• This is expected! Forked PRs don't have access to contexts
• Security scans will be skipped or fail due to missing credentials
• After code review, maintainers can manually trigger with secrets

Publishing Fails

• Verify PYPI_TOKEN is set correctly in the pypi-publish context
• Check that you're on the master branch
• Ensure version in setup.py hasn't been published before

Security Scans Fail

• Check context configurations (static-analysis, okta-dcp)
• Verify tokens and credentials are valid
• Review security scan logs for specific issues

Build Fails

• Check Python version compatibility
• Verify all dependencies in requirements.txt are available
• Review test failures in CircleCI artifacts

Best Practices

1. Always create feature branches: Don't commit directly to master
2. Wait for CI to pass: Ensure all checks pass on feature branches before merging
3. Review security scan results: Address vulnerabilities before merging to master
4. Use semantic versioning: Tag releases with v prefix (e.g., v3.0.1)
5. Update version in setup.py: Before merging to master for a release

Pipeline Diagram

Non-Prod (Internal PRs and Feature Branches)

┌─────────────────────────────────────────────────────────────┐
│            Non-Prod: Internal Contributors                   │
├─────────────────────────────────────────────────────────────┤
│  Step 1: Security Scans (Run in Parallel)                   │
│  ┌─────────────────────┐  ┌─────────────────────┐          │
│  │  snyk-scan          │  │  reversing-labs     │          │
│  │  • Vuln scanning    │  │  • Malware scan     │          │
│  └─────────────────────┘  └─────────────────────┘          │
│           │                         │                        │
│           └────────┬────────────────┘                        │
│                    ↓                                         │
│  Step 2: Build (Only if security scans pass)                │
│  ┌────────────────────────────────────────┐                 │
│  │  build                                 │                 │
│  │  • Install dependencies                │                 │
│  │  • Build distribution packages         │                 │
│  └────────────────────────────────────────┘                 │
│                    ↓                                         │
│  Step 3: Unit Tests (Only if build succeeds)                │
│  ┌────────────────────────────────────────┐                 │
│  │  unit_tests                            │                 │
│  │  • Run tests/unit                      │                 │
│  └────────────────────────────────────────┘                 │
│                    ↓                                         │
│  Step 4: Integration Tests (Only if unit tests pass)        │
│  ┌────────────────────────────────────────┐                 │
│  │  integration_tests                     │                 │
│  │  • Run tests/integration               │                 │
│  └────────────────────────────────────────┘                 │
└─────────────────────────────────────────────────────────────┘

### Contributors (External/Forked Repository PRs)

┌─────────────────────────────────────────────────────────────┐
│ Contributors: External Contributors │
├─────────────────────────────────────────────────────────────┤
│ ⚠️ Security scans skipped (no context access) │
│ │
│ Step 1: Build │
│ ┌────────────────────────────────────────┐ │
│ │ build │ │
│ │ • Install dependencies │ │
│ │ • Build distribution packages │ │
│ └────────────────────────────────────────┘ │
│ ↓ │
│ Step 2: Unit Tests (Only if build succeeds) │
│ ┌────────────────────────────────────────┐ │
│ │ unit_tests │ │
│ │ • Run tests/unit │ │
│ └────────────────────────────────────────┘ │
│ ↓ │
│ Step 3: Integration Tests (Only if unit tests pass) │
│ ┌────────────────────────────────────────┐ │
│ │ integration_tests │ │
│ │ • Run tests/integration │ │
│ └────────────────────────────────────────┘ │
│ │
│ ⚠️ Maintainer should review code before merging │
└─────────────────────────────────────────────────────────────┘

Prod (Master Branch Only + Version Tags)

┌─────────────────────────────────────────────────────────────┐
│                  Prod: Master Branch Only                    │
├─────────────────────────────────────────────────────────────┤
│  Step 1: Security Scans (Run in Parallel)                   │
│  ┌─────────────────────┐  ┌─────────────────────┐          │
│  │  snyk-scan          │  │  reversing-labs     │          │
│  │  • Vuln scanning    │  │  • Malware scan     │          │
│  └─────────────────────┘  └─────────────────────┘          │
│           │                         │                        │
│           └────────┬────────────────┘                        │
│                    ↓                                         │
│  Step 2: Build (Only if security scans pass)                │
│  ┌────────────────────────────────────────┐                 │
│  │  build                                 │                 │
│  │  • Install dependencies                │                 │
│  │  • Build distribution packages         │                 │
│  └────────────────────────────────────────┘                 │
│                    ↓                                         │
│  Step 3: Unit Tests (Only if build succeeds)                │
│  ┌────────────────────────────────────────┐                 │
│  │  unit_tests                            │                 │
│  │  • Run tests/unit                      │                 │
│  └────────────────────────────────────────┘                 │
│                    ↓                                         │
│  Step 4: Integration Tests (Only if unit tests pass)        │
│  ┌────────────────────────────────────────┐                 │
│  │  integration_tests                     │                 │
│  │  • Run tests/integration               │                 │
│  └────────────────────────────────────────┘                 │
│                    ↓                                         │
│  Step 5: Publish (Only if all tests succeed)                │
│  ┌────────────────────────────────────────┐                 │
│  │  publish_to_pypi                       │                 │
│  │  • Verify distribution                 │                 │
│  │  • Upload to PyPI                      │                 │
│  └────────────────────────────────────────┘                 │
└─────────────────────────────────────────────────────────────┘

Notes

• The pipeline uses dependency caching to speed up builds
• Test results are stored as artifacts for review
• Distribution files are verified before publishing
• All jobs use Python 3.10 Docker image for consistency