Seed Consistency Issues Due to Non-Deterministic Flatland Line Generator

[RobinU434]

Summary

The SwitchFL environment experiences seed consistency issues that prevent reproducible training runs and deterministic testing. Despite setting the same random seed, identical environments produce different train configurations, leading to divergent behavior during simulation steps.

Problem Description

Expected Behavior

When creating two SwitchFL environments with identical parameters and the same seed, they should:

• Have trains at identical positions with identical directions
• Produce identical observations for the same agents
• Follow identical step sequences when given the same actions
• Enable reproducible reinforcement learning experiments

Actual Behavior

Even with identical seeds, environments exhibit:

• Non-deterministic train direction assignment - same positions, different directions
• Inconsistent active agent scheduling - different switches become active first
• Divergent step sequences - identical actions lead to different outcomes
• Non-reproducible training runs - impossible to replicate experimental results

Root Cause Analysis

Through extensive investigation, we identified that Flatland's sparse_line_generator has inherent non-determinism that cannot be controlled through standard Python seeding mechanisms.

Evidence

1. Train Direction Randomness

# Same seed (42), different runs produce different directions:
# Run 1: Train 0 at (5,5) direction=NORTH, Train 1 at (20,5) direction=WEST  
# Run 2: Train 0 at (5,5) direction=SOUTH, Train 1 at (20,5) direction=EAST
# Run 3: Train 0 at (5,5) direction=NORTH, Train 1 at (20,5) direction=WEST

2. Position Consistency but Direction Inconsistency

# Positions are deterministic, directions are not:
env1.reset(seed=999)  # Train at (5,5) going NORTH → activates switch_4-5
env2.reset(seed=999)  # Train at (5,5) going SOUTH → activates switch_6-5

3. Step-Level Divergence

# First active agents differ due to train direction differences:
Environment 1: switch_4-5 (train going NORTH from (5,5))
Environment 2: switch_6-5 (train going SOUTH from (5,5))

Technical Details

Affected Components

• File: project/environment/switchfl/switch_env.py - Environment reset and step logic
• File: project/environment/test/test_seed_consistency.py - Failing seed consistency tests
• Root Cause: Flatland's sparse_line_generator in train placement logic

Failed Test Cases

1. test_environment_step_consistency - Active agents differ at step 0
2. test_seed_consistency_summary - Train direction assignments inconsistent
3. Multiple other seed-dependent tests showing partial consistency

Seeding Attempts Made

We attempted comprehensive seeding at multiple levels:

# Environment level seeding
np.random.seed(seed)
random.seed(seed)

# Flatland specific seeding  
rail_generator = sparse_rail_generator(..., random_seed=seed)
line_generator = sparse_line_generator(..., random_seed=seed)

# Complete state reset
import os
os.environ['PYTHONHASHSEED'] = str(seed)
torch.manual_seed(seed) if torch.available
tf.random.set_seed(seed) if tf available

Result: None of these approaches eliminated the non-determinism.

Impact

For Developers

• Unreproducible Experiments: Cannot replicate training results
• Flaky Tests: Seed-dependent tests fail intermittently
• Debugging Difficulties: Cannot create identical scenarios for investigation
• Research Validity: Results cannot be reproduced by other researchers

For Users

• Training Inconsistency: Same hyperparameters produce different results
• Model Comparison Issues: Cannot fairly compare different approaches
• Deployment Uncertainty: Model behavior may vary between runs

Reproduction Steps

1. Clone the repository:

   git clone <repository-url>
   cd DistrQLearn/project/environment

2. Run the seed consistency test:

   python -m pytest test/test_seed_consistency.py::TestSeedConsistency::test_environment_step_consistency -v

3. Expected failure:

   ValueError: Active agents differ, cannot proceed with step consistency test
   Environment 1: switch_4-5
   Environment 2: switch_6-5
   

4. Alternative reproduction script:

   from switchfl.utils.build_env import build_standard_async_env
   
   # Create two identical environments
   env1 = build_standard_async_env(height=25, width=25, num_trains=2, seed=42)
   env2 = build_standard_async_env(height=25, width=25, num_trains=2, seed=42)
   
   env1.reset(seed=42)
   env2.reset(seed=42)
   
   # Check if first active agents match
   agent1 = next(env1.agent_iter())  
   agent2 = next(env2.agent_iter())
   print(f"Agent 1: {agent1}, Agent 2: {agent2}")  # Will likely differ

Environment Information

• Python Version: 3.12
• Flatland-RL Version: (current version in requirements)
• Operating System: Linux (Ubuntu/similar)
• Hardware: Any (issue is algorithmic, not hardware-dependent)

Proposed Solutions

Short-term Workarounds

1. Accept Partial Consistency: Focus on components that are deterministic (rail topology, action spaces)
2. Statistical Testing: Use ensemble averaging instead of single-run reproducibility
3. Relaxed Test Assertions: Make tests more tolerant of expected non-determinism

Long-term Solutions

1. Custom Line Generator: Implement a fully deterministic train placement algorithm
2. Flatland Fork: Modify Flatland's line generator to be fully deterministic
3. Environment Cloning: Create environments by copying/cloning instead of regenerating
4. Post-Generation Sorting: Enforce deterministic train ordering after Flatland generation

Recommended Approach

We recommend implementing Solution 1: Accept Partial Consistency for immediate needs, while working toward Solution 4: Post-Generation Sorting as a medium-term fix.

Partial Fix Implemented

We have implemented a partial fix that improves consistency by deterministically reassigning train handles after environment creation:

# In switch_env.py reset() method
if seed is not None and hasattr(self.rail_env, 'agents') and len(self.rail_env.agents) > 1:
    # Sort agents by initial position to ensure consistent ordering
    sorted_agents = sorted(self.rail_env.agents, 
                          key=lambda agent: (agent.initial_position or (0, 0)))
    
    # Reassign handles deterministically
    for new_handle, agent in enumerate(sorted_agents):
        agent.handle = new_handle
    
    self.rail_env.agents = sorted_agents

Result: This fixes train handle consistency and improves most seed consistency tests, but train direction randomness remains.

Additional Context

This issue affects the scientific rigor of experiments using SwitchFL. While the environment functions correctly for training, the lack of reproducibility limits research validity and makes debugging significantly more challenging.

The issue appears to be fundamental to Flatland's design and may require upstream fixes or local workarounds, depending on the criticality of reproducibility for specific use cases.