Critical Enhancements & Bug Fixes

.github/workflows/ci.yml

@@ -0,0 +1,122 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.24'
+
+      - name: Download dependencies
+        run: go mod download
+
+      - name: Run tests
+        # Note: -race disabled due to pre-existing race conditions in tests that spawn
+        # background goroutines. These should be fixed by adding proper synchronization.
+        run: go test -v ./...
+
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.24'
+
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v6
+        with:
+          version: latest
+
+  security:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.24'
+
+      - name: Run gosec
+        uses: securego/gosec@master
+        with:
+          # Exclude G101 (hardcoded credentials - false positive on env var names)
+          # Exclude G115 (integer overflow - false positive for PR numbers)
+          # Exclude G304 (file inclusion - intentional for CLI tools)
+          # Exclude G306 (file permissions - config files don't need 0600)
+          args: -exclude=G101,G115,G304,G306 ./...
+
+  build:
+    runs-on: ubuntu-latest
+    needs: [test, lint]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.24'
+
+      - name: Build
+        run: go build -v ./...
+
+  deploy:
+    runs-on: ubuntu-latest
+    needs: [build, security]
+    # Only deploy on push to main (not on PRs)
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+
+    permissions:
+      contents: read
+      id-token: write  # Required for Workload Identity Federation
+
+    env:
+      PROJECT_ID: "github-copy-code-examples"
+      SERVICE_NAME: "examples-copier"
+      REGION: "us-central1"
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Authenticate to Google Cloud
+        uses: google-github-actions/auth@v2
+        with:
+          workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
+          service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
+
+      - name: Set up Cloud SDK
+        uses: google-github-actions/setup-gcloud@v2
+
+      - name: Deploy to Cloud Run
+        run: |
+          gcloud run deploy $SERVICE_NAME \
+            --source . \
+            --region $REGION \
+            --project $PROJECT_ID \
+            --allow-unauthenticated \
+            --env-vars-file=env-cloudrun.yaml \
+            --max-instances=10 \
+            --cpu=1 \
+            --memory=512Mi \
+            --timeout=300s \
+            --concurrency=80 \
+            --port=8080 \
+            --platform=managed
+
+      - name: Show deployment URL
+        run: |
+          URL=$(gcloud run services describe $SERVICE_NAME \
+            --region $REGION \
+            --project $PROJECT_ID \
+            --format='value(status.url)')
+          echo "🚀 Deployed to: $URL"
+

.gitignore

@@ -1,5 +1,5 @@
 # Binaries
-examples-copier
+github-copier
 code-copier
 copier
 *.exe
@@ -23,19 +23,21 @@ go.work
 # Environment files with secrets (working files - create from templates in configs/)
 # Working files (create from templates):
 #   env.yaml - App Engine deployment config (from configs/env.yaml.*)
-#   env-cloudrun.yaml - Cloud Run deployment config (from configs/env.yaml.*)
 #   .env - Local development config (from configs/.env.local.example)
+#   .env.test - Test config (from testdata/.env.test)
+# Note: env-cloudrun.yaml is committed (contains no secrets, only Secret Manager references)
 env.yaml
-env-cloudrun.yaml
 .env
 .env.local
+.env.test
 .env.production
 .env.*.local
 
-# Explicitly keep template files in configs/ directory (these should be tracked)
+# Explicitly keep template files (these should be tracked)
 !configs/env.yaml.example
 !configs/env.yaml.production
 !configs/.env.local.example
+!testdata/.env.test
 
 # Private keys
 *.pem
@@ -58,3 +60,4 @@ Thumbs.db
 # Temporary files
 tmp/
 temp/
+RECOMMENDATIONS.md

.pre-commit-config.yaml

@@ -0,0 +1,36 @@
+repos:
+  # Secrets detection
+  - repo: https://github.com/gitleaks/gitleaks
+    rev: v8.21.2
+    hooks:
+      - id: gitleaks
+
+  # Go linting
+  - repo: https://github.com/golangci/golangci-lint
+    rev: v1.62.2
+    hooks:
+      - id: golangci-lint
+
+  # Local Go hooks
+  - repo: local
+    hooks:
+      - id: go-fmt
+        name: go fmt
+        entry: gofmt -w
+        language: system
+        types: [go]
+
+      - id: go-vet
+        name: go vet
+        entry: go vet ./...
+        language: system
+        pass_filenames: false
+        types: [go]
+
+      - id: go-build
+        name: go build
+        entry: go build ./...
+        language: system
+        pass_filenames: false
+        types: [go]
+

AGENT.md

@@ -0,0 +1,90 @@
+# Agent Context: GitHub Copier
+
+Webhook service: PR merged → match files → transform paths → copy to target repos.
+
+## File Map
+
+```
+app.go                              # entrypoint, HTTP server
+services/
+  webhook_handler_new.go            # HandleWebhookWithContainer()
+  workflow_processor.go             # ProcessWorkflow() - core logic
+  pattern_matcher.go                # MatchFile(pattern, path) bool
+  github_auth.go                    # ConfigurePermissions() error
+  github_read.go                    # GetFilesChangedInPr(), RetrieveFileContents()
+  github_write_to_target.go         # AddFilesToTargetRepoBranch()
+  github_write_to_source.go         # UpdateDeprecationFile()
+  file_state_service.go             # tracks upload/deprecate queues
+  main_config_loader.go             # LoadConfig() with $ref support
+  service_container.go              # DI container
+types/
+  config.go                         # Workflow, Transformation, SourcePattern structs
+  types.go                          # ChangedFile, UploadKey, UploadFileContent
+configs/environment.go              # Config struct, LoadEnvironment()
+tests/utils.go                      # test helpers, httpmock setup
+```
+
+## Key Types
+
+```go
+// types/config.go
+type PatternType string    // "prefix" | "glob" | "regex"
+type TransformationType string  // "move" | "copy" | "glob" | "regex"
+
+type Workflow struct {
+    Name           string
+    Source         SourceConfig      // Repo, Branch, Patterns []SourcePattern
+    Destination    DestinationConfig // Repo, Branch
+    Transformations []Transformation // Type, From, To, Pattern, Replacement
+    Commit         CommitConfig      // Strategy, Message, PRTitle, AutoMerge
+}
+
+// types/types.go
+type ChangedFile struct { Path, Status string }  // Status: "ADDED"|"MODIFIED"|"DELETED"
+type UploadKey struct { RepoName, BranchPath string }
+```
+
+## Global State (⚠️ mutable)
+
+```go
+// services/github_write_to_target.go
+var FilesToUpload map[UploadKey]UploadFileContent
+// services/github_auth.go
+var InstallationAccessToken string
+var OrgTokens map[string]string
+```
+
+## Config Example
+
+```yaml
+workflows:
+  - name: "sync-docs"
+    source: { repo: "org/src", branch: "main", patterns: [{type: glob, pattern: "docs/**"}] }
+    destination: { repo: "org/dest", branch: "main" }
+    transformations: [{ type: move, from: "docs/", to: "public/" }]
+    commit: { strategy: pr, message: "Sync" }  # strategy: direct|pr
+```
+
+## Test Commands
+
+```bash
+go test ./...                                    # all
+go test ./services/... -run TestWorkflow -v      # specific
+```
+
+## Edit Patterns
+
+| Task | Files to modify |
+|------|-----------------|
+| New transformation | `types/config.go` (TransformationType) → `workflow_processor.go` (processFileForWorkflow) |
+| New pattern type | `types/config.go` (PatternType) → `pattern_matcher.go` |
+| New config field | `types/config.go` (struct) → consumers in `workflow_processor.go` |
+| Webhook logic | `webhook_handler_new.go` |
+
+## Conventions
+
+- Return `error`, never `log.Fatal`
+- Wrap errors: `fmt.Errorf("context: %w", err)`
+- Nil-check GitHub API responses before dereference
+- Tests use `httpmock`; see `tests/utils.go`
+- **Changelog**: Update `CHANGELOG.md` for all notable changes (follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/))

CHANGELOG.md

@@ -0,0 +1,54 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## 17 Dec 2025
+
+### Added
+- CI/CD pipeline with GitHub Actions (`.github/workflows/ci.yml`)
+  - Test job
+  - Lint job with golangci-lint
+  - Security scanning with gosec
+  - Build verification
+  - Automated deployment to Cloud Run on merge to main (via Workload Identity Federation)
+- Pre-commit hooks for secrets detection and Go linting (`.pre-commit-config.yaml`)
+- AGENT.md for AI agent context
+- Comprehensive test suite for `workflow_processor.go` (843 lines, 94%+ coverage)
+- Integration test harness for local testing (`scripts/integration-test.sh`)
+- Test environment configuration (`testdata/.env.test`)
+
+### Changed
+- Renamed module from `github.com/mongodb/code-example-tooling/code-copier` to `github.com/grove-platform/github-copier`
+- Renamed binary from `examples-copier` to `github-copier`
+- Renamed `test-payloads/` to `testdata/` (Go convention)
+- All `log.Fatal` calls replaced with proper error returns for graceful error handling
+- `FileStateService.filesToDeprecate` changed from single-entry map to slice-based accumulation
+
+### Fixed
+- Deprecation file accumulation bug: multiple deprecated files now correctly accumulate instead of overwriting
+- Nil pointer dereference bugs across GitHub API calls in:
+  - `services/github_read.go`
+  - `services/github_write_to_source.go`
+  - `services/main_config_loader.go`
+  - `services/config_loader.go`
+- DELETED file status handling: GitHub GraphQL API returns uppercase `DELETED` but code checked for lowercase `removed`
+- Graceful shutdown now properly waits for in-flight requests and cleans up resources
+
+### Security
+- Added gitleaks pre-commit hook for secrets detection
+- Added gosec security scanning in CI pipeline
+
+## Initial Release (Migration from mongodb/code-example-tooling)
+
+### Features
+- Webhook service for automated file copying on PR merge
+- Pattern matching support: prefix, glob, regex
+- Transformation types: move, copy, glob, regex
+- Main config system with `$ref` support for distributed workflow configs
+- Commit strategies: direct commit or pull request
+- Health and metrics endpoints
+- Slack notifications for operational visibility
+- MongoDB audit logging (optional)
+- Google Cloud Logging integration
+- Dry-run mode for testing
+