feat: Add OpenTelemetry instrumentation for event handlers

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>

Author: yordis

OTEL_PR_SPLIT_PLAN.md

@@ -0,0 +1,347 @@
+# OpenTelemetry Integration - PR Split Plan
+
+## Overview
+
+This document outlines how to split the OpenTelemetry integration into multiple, reviewable PRs. The goal is to:
+1. Get a minimal working version merged first (MVP) - **each PR must deliver actual tracing functionality**
+2. Incrementally add features - **no PR should be just helpers/attributes without instrumentation**
+3. Keep each PR focused and reviewable - **each PR should create real spans that users can see**
+
+**Key Principle**: Every PR must include instrumentation that creates actual spans. Helpers and attributes are only included when they're needed by the instrumentation in the same PR.
+
+## PR Summary
+
+| PR | What It Delivers | Spans Created | Status |
+|----|------------------|---------------|--------|
+| **PR 1** | Event handler tracing | `commanded.event.handle`<br>`commanded.event.batch` | 🔴 MVP - Simpler, can work standalone |
+| **PR 2** | Command → Aggregate tracing | `commanded.application.dispatch`<br>`commanded.aggregate.execute` | 🟡 High priority - Completes flow |
+| **PR 3** | Aggregate population tracing | `commanded.aggregate.populate` | 🟢 Nice-to-have |
+| **PR 4** | Event store operations | `commanded.event_store.*` | 🟢 Optional (disabled by default) |
+| **PR 5** | Documentation | N/A | 📚 Final step |
+
+---
+
+## PR 1: Event Handler Tracing (MVP - CRITICAL) 🔴
+
+**Goal**: Enable event handler tracing - simpler starting point, can work standalone
+
+**Status**: Must merge first - simpler implementation, establishes foundation
+
+**Rationale**: Event handlers are simpler (only need metadata, no execution_context) and can work independently. This establishes the infrastructure (helpers, attributes, setup) that PR 2 will reuse.
+
+### Tasks Checklist
+
+#### Core Setup & Configuration (required for instrumentation)
+- [ ] Create `lib/commanded/opentelemetry.ex` - Main setup module with validation
+- [ ] Create `lib/commanded/opentelemetry/helper.ex` - Context propagation helpers (used by instrumentation)
+- [ ] Create `lib/commanded/opentelemetry/attributes.ex` - Attribute builders (used by instrumentation)
+
+#### Instrumentation (THE MEAT - this is what makes the PR meaningful)
+- [ ] Create `lib/commanded/opentelemetry/event_handler.ex` - Single + batch event handler tracing
+
+#### Test Infrastructure
+- [ ] Create `test/support/opentelemetry_case.ex` - Test case template
+- [ ] Create `test/opentelemetry/helper_test.exs` - Helper function tests
+- [ ] Create `test/opentelemetry/options_test.exs` - Configuration validation tests
+- [ ] Create `test/opentelemetry/opentelemetry_setup_test.exs` - Setup tests
+- [ ] Create `test/opentelemetry/event_handler_test.exs` - Event handler tests
+
+#### Dependencies & Configuration
+- [ ] Add `opentelemetry_telemetry` to `mix.exs` (optional dependency)
+- [ ] Add `opentelemetry_semantic_conventions` to `mix.exs` (optional dependency)
+- [ ] Add `nimble_options` to `mix.exs` (optional dependency)
+- [ ] Add `Commanded.OpenTelemetry` to docs extras in `mix.exs`
+- [ ] Add `Commanded.OpenTelemetry` to `nest_modules_by_prefix` in `mix.exs`
+- [ ] Add `opentelemetry_telemetry` to dialyzer plt_add_apps in `mix.exs`
+
+### What This Delivers (Real Functionality)
+
+**Primary Value:**
+- ✅ **`commanded.event.handle` spans** - Actual tracing of single event handling
+- ✅ **`commanded.event.batch` spans** - Actual tracing of batch event handling
+- ✅ Event handler visibility (can work standalone)
+- ✅ Exception tracking in event handlers
+- ✅ Span linking/child relationships based on `span_relationship` config
+- ✅ Trace context extraction from metadata (if traceparent present)
+
+**Supporting Infrastructure (only included because instrumentation needs it):**
+- ✅ `Commanded.OpenTelemetry.setup/0` function
+- ✅ Option validation with NimbleOptions
+- ✅ Helper functions (used by instrumentation modules)
+- ✅ Attribute builders (used by instrumentation modules)
+- ✅ Test infrastructure (validates the functionality)
+
+### Spans Created
+
+1. **`commanded.event.handle`**
+   - Kind: `:consumer`
+   - Attributes: event, event_id, handler_name, stream_id, etc.
+   - Span relationship: Configurable (`:link`, `:child`, `:none`)
+   - Created when: Single event is handled
+
+2. **`commanded.event.batch`**
+   - Kind: `:consumer`
+   - Attributes: batch_count, handler_name
+   - Span relationship: Always `:link` (cannot be child of multiple commands)
+   - Created when: Batch of events is handled
+
+### Verification Checklist
+
+- [ ] `Commanded.OpenTelemetry.setup()` can be called without errors
+- [ ] Options are validated correctly (tested in `options_test.exs`)
+- [ ] **Spans created for single event handling** (verified in `event_handler_test.exs`)
+- [ ] **Spans created for batch event handling** (verified in `event_handler_test.exs`)
+- [ ] **Span links work** (`:link` mode) - tested with traceparent in metadata
+- [ ] **Span children work** (`:child` mode) - tested with parent span context
+- [ ] **Exceptions recorded properly** (tested in event handler exception test)
+- [ ] All attributes present and correct
+- [ ] All tests pass: `mix test test/opentelemetry/`
+- [ ] Code compiles without warnings: `mix compile`
+
+---
+
+## PR 2: Command Dispatch & Aggregate Execution Tracing 🟡
+
+**Goal**: Complete end-to-end tracing with command → aggregate visibility
+
+**Status**: High priority - completes the full flow
+
+**Depends on**: PR 1 (reuses helpers, attributes, setup infrastructure)
+
+### Tasks Checklist
+
+#### Instrumentation (THE MEAT)
+- [ ] Create `lib/commanded/opentelemetry/application.ex` - Command dispatch tracing
+- [ ] Create `lib/commanded/opentelemetry/aggregate.ex` - Aggregate execution tracing
+  - Uses helpers and attributes from PR 1 (no new infrastructure needed)
+
+#### Tests
+- [ ] Create `test/opentelemetry/application_test.exs` - Command dispatch tests
+- [ ] Create `test/opentelemetry/aggregate_test.exs` - Aggregate execution tests
+- [ ] Create `test/opentelemetry/trace_context_propagator_e2e_test.exs` - Full flow E2E test
+
+### What This Delivers (Real Functionality)
+
+- ✅ **`commanded.application.dispatch` spans** - Actual tracing of command dispatch
+- ✅ **`commanded.aggregate.execute` spans** - Actual tracing of aggregate execution
+- ✅ Full command → aggregate trace visibility
+- ✅ Exception tracking in aggregates
+- ✅ Parent-child span relationships (aggregate is child of dispatch)
+- ✅ Completes end-to-end tracing: command → aggregate → event handler (with PR 1)
+
+### Spans Created
+
+1. **`commanded.application.dispatch`**
+   - Kind: `:producer`
+   - Attributes: application, command, causation_id, correlation_id
+   - Created when: Command is dispatched
+
+2. **`commanded.aggregate.execute`**
+   - Kind: `:consumer`
+   - Attributes: application, aggregate_uuid, aggregate_version, command
+   - Created when: Aggregate executes command
+   - Includes: Exception tracking, event count
+
+### Critical Feature: Completes Trace Flow
+
+This PR completes the end-to-end trace:
+- Command dispatch creates producer span
+- Aggregate execution creates child consumer span
+- Event handlers (from PR 1) can link to these spans via trace context
+
+### Verification Checklist
+
+- [ ] **Spans created for command dispatch** (verified in `application_test.exs`)
+- [ ] **Spans created for aggregate execution** (verified in `aggregate_test.exs`)
+- [ ] **Spans linked correctly** (aggregate is child of dispatch)
+- [ ] **Exceptions recorded properly** (tested in aggregate exception test)
+- [ ] Trace context propagation works (verified in E2E test with PR 1)
+- [ ] All attributes present and correct
+- [ ] All tests pass: `mix test test/opentelemetry/application_test.exs test/opentelemetry/aggregate_test.exs test/opentelemetry/trace_context_propagator_e2e_test.exs`
+- [ ] Code compiles without warnings: `mix compile`
+
+---
+
+## PR 3: Aggregate Population Tracing 🟢
+
+**Goal**: Trace aggregate loading from events
+
+**Status**: Nice-to-have - useful for performance analysis
+
+**Depends on**: PR 1
+
+### Tasks Checklist
+
+#### Instrumentation
+- [ ] Create `lib/commanded/opentelemetry/aggregate_populate.ex`
+  - Uses helpers and attributes from PR 1 (no new infrastructure needed)
+
+#### Tests
+- [ ] Add tests to `test/opentelemetry/aggregate_test.exs` OR create new test file `test/opentelemetry/aggregate_populate_test.exs`
+
+### What This Delivers (Real Functionality)
+
+- ✅ **`commanded.aggregate.populate` spans** - Actual tracing of aggregate loading
+- ✅ Visibility into aggregate loading performance
+- ✅ Event count tracking during population
+
+### Spans Created
+
+1. **`commanded.aggregate.populate`**
+   - Kind: `:internal`
+   - Attributes: aggregate_uuid, aggregate_version, event_count
+   - Created when: Aggregate is loaded from events
+
+### Verification Checklist
+
+- [ ] Spans created during aggregate population (verified in tests)
+- [ ] Event count attribute set correctly in span
+- [ ] Aggregate version tracked in span attributes
+- [ ] All tests pass: `mix test test/opentelemetry/aggregate*test.exs`
+- [ ] Code compiles without warnings: `mix compile`
+
+---
+
+## PR 4: Event Store Operations Tracing 🟢
+
+**Goal**: Low-level event store operation visibility
+
+**Status**: Optional - disabled by default
+
+**Depends on**: PR 1
+
+### Tasks Checklist
+
+#### Instrumentation
+- [ ] Create `lib/commanded/opentelemetry/event_store.ex`
+  - Uses helpers and attributes from PR 1 (no new infrastructure needed)
+  - Handles all event store operations: append_to_stream, stream_forward, subscribe_to, ack_event, etc.
+
+#### Tests
+- [ ] Create `test/opentelemetry/event_store_test.exs`
+
+### What This Delivers (Real Functionality)
+
+- ✅ **`commanded.event_store.*` spans** - Actual tracing of event store operations
+- ✅ Low-level debugging visibility
+- ✅ Performance analysis of event store operations
+
+### Spans Created
+
+Multiple spans for event store operations:
+- `commanded.event_store.append_to_stream`
+- `commanded.event_store.stream_forward`
+- `commanded.event_store.subscribe_to`
+- `commanded.event_store.ack_event`
+- And more...
+
+### Configuration
+
+- **Disabled by default** (`event_store: false`)
+- Enable with: `Commanded.OpenTelemetry.setup(tracing: [event_store: true])`
+
+### Verification Checklist
+
+- [ ] All event store operations traced (append_to_stream, stream_forward, subscribe_to, ack_event, etc.)
+- [ ] Disabled by default (`event_store: false` in default config)
+- [ ] Can be enabled via config: `Commanded.OpenTelemetry.setup(tracing: [event_store: true])`
+- [ ] All attributes present and correct
+- [ ] All tests pass: `mix test test/opentelemetry/event_store_test.exs`
+- [ ] Code compiles without warnings: `mix compile`
+
+---
+
+## PR 5: Documentation 📚
+
+**Goal**: User-facing documentation
+
+**Status**: Final step
+
+**Depends on**: PRs 1-2 (minimum), ideally PRs 1-4
+
+### Tasks Checklist
+
+#### Documentation Files
+- [ ] Create/update `guides/howtos/setting-up-opentelemetry-tracing.md` - Complete user guide
+- [ ] Update `guides/explanations/fork-differences.md` - Add OpenTelemetry section
+
+#### Documentation Content
+- [ ] Write installation instructions
+- [ ] Write setup examples (basic and advanced)
+- [ ] Document all configuration options
+- [ ] Document span relationship modes (`:link`, `:child`, `:none`)
+- [ ] Write troubleshooting section
+- [ ] Add examples with Jaeger/OTLP exporters
+- [ ] Document middleware usage (`Commanded.Middleware.TraceContextPropagator`)
+
+### Verification Checklist
+
+- [ ] Complete setup guide with working examples
+- [ ] All configuration options documented with examples
+- [ ] Examples tested and verified to work
+- [ ] Troubleshooting section includes common issues
+- [ ] Documentation builds successfully: `mix docs`
+
+---
+
+## Critical Path Summary
+
+### Minimum Viable Product (MVP)
+
+**PR 1** → **PR 2**
+
+This gives you:
+- ✅ Event handler tracing (PR 1 - can work standalone)
+- ✅ Full command → aggregate → event handler tracing (PR 1 + PR 2)
+- ✅ Trace context propagation
+- ✅ End-to-end visibility
+
+### Full Feature Set
+
+**PR 1** → **PR 2** → **PR 3** → **PR 4** → **PR 5**
+
+---
+
+## Notes
+
+### Middleware
+
+**Decision**: Use `Commanded.Middleware.TraceContextPropagator` directly.
+
+- All documentation references `Commanded.Middleware.TraceContextPropagator`
+- No alias module needed - keeps the codebase simpler
+- The middleware already exists and works correctly
+- Users can add it to their routers as: `middleware Commanded.Middleware.TraceContextPropagator`
+
+### Testing Strategy
+
+- **PR 1**: Includes test infrastructure AND event handler tests (validates real spans, simpler to test)
+- **PR 2**: Includes command/aggregate tests and E2E validation (validates real spans, completes flow)
+- **PR 3+**: Can reuse test infrastructure from PR 1 (validates real spans)
+- Each PR must demonstrate actual functionality with passing tests
+
+### Why Event Handlers First?
+
+**Advantages:**
+- Simpler implementation (only needs metadata, no execution_context)
+- Can work standalone (event handlers process events independently)
+- Establishes infrastructure (helpers, attributes, setup) that PR 2 reuses
+- Easier to test (just need event metadata)
+
+**Trade-off:**
+- Event handler spans may be orphaned until PR 2 (but still useful for debugging event processing)
+- Full trace flow only visible after PR 2 (but PR 1 provides immediate value)
+
+### Breaking Changes
+
+- None expected - all optional dependencies
+- All features opt-in via `setup()` call
+- Defaults are sensible (event_store disabled)
+
+### Migration Notes
+
+The middleware `Commanded.Middleware.TraceContextPropagator` is used directly:
+- Already exists and works correctly
+- No migration needed - it's the same module
+- Simply add to router: `middleware Commanded.Middleware.TraceContextPropagator`
+

guides/explanations/fork-differences.md

@@ -171,3 +171,21 @@ end
 - Uses standard W3C `traceparent` and `tracestate` headers stored in event metadata
 - Event handlers can extract trace context to create span links or parent-child relationships
 - Non-invasive - only adds metadata when a span is active
+
+### **OpenTelemetry Integration - Event Handler Tracing**
+[PR #41](https://github.com/straw-hat-team/commanded/pull/41) (Part 1 of multi-PR effort)
+
+**What Changed:**
+Added OpenTelemetry distributed tracing support for event handlers. This enables observability of event processing operations in distributed systems.
+
+**Why:**
+Event handlers run asynchronously across process boundaries, making it difficult to correlate event processing with the commands that triggered them. OpenTelemetry integration provides distributed tracing capabilities to visualize the complete request flow from command dispatch through event handling.
+
+**Implementation Approach:**
+- Uses Commanded's existing `:telemetry` events to create OpenTelemetry spans non-invasively
+- Integrates with `opentelemetry_telemetry` bridge library
+- Leverages W3C Trace Context propagation middleware to maintain trace correlation across processes
+- Follows OpenTelemetry semantic conventions for messaging systems
+- Supports configurable span relationships (`:link`, `:child`, `:none`) to handle different event processing patterns
+
+**Note:** This is the first PR in a planned series to add comprehensive OpenTelemetry instrumentation. Future PRs will add tracing for command dispatch, aggregate execution, and event store operations.