Enable realtime pipeline execution and improve data fetching

[pramilajangid]

Fixes #16

Summary

• add Docker and runtime setup for local and containerized execution
• improve EIA and weather download handling for realtime runs
• fix path and config issues across realtime pipeline stages
• update first-tier and second-tier forecasting paths for realtime execution
• validate realtime runs for AECI and PJM with exact-date weather data

Blockers Fixed

• Broken Docker setup: docker-compose.yml pointed at a missing Dockerfile, so I added a working Dockerfile and fixed Compose to build the app again.
• Docker dependency mismatch: the container only installed API requirements, so pipeline imports like pandas failed; I updated Docker to install both root and API requirements.
• Apple Silicon / cross-platform Docker breakage: old TensorFlow wheels were unreliable on native arm64, so Docker now defaults to configurable linux/amd64 for safer Windows/Mac portability.
• Invalid dependency pin: requirements.txt had xarray==23.7.0, which does not exist; I corrected it to 2023.7.0 so installs succeed.
• Hardcoded / unsafe EIA access: src/eiaParser.py assumed happy-path API responses and leaked the key in logs; I moved the key to EIA_API_KEY, validated responses, and masked the key in printed URLs.
• EIA fetch crashing the whole pipeline: live electricity download failures used to abort everything; I added controlled EIA errors plus cached-electricity fallback in src/CarbonCastE2E.py.
• Relative path bugs in realtime execution: several pipeline stages assumed the process was launched from src/; I converted electricity, weather, first-tier, and second-tier paths to resolve from the repo/source location instead.
• Weather downloader writing invalid files: src/weather/getRealTimeWeatherData.py could save HTML/error pages as .grib2; I added HTTP status checks, empty/HTML rejection, and explicit weather download errors.
• Weather output directory missing: realtime weather download/parsing failed because target folders did not exist; I added directory creation before writes.
• Wrong-date weather fallback risk: the pipeline could have drifted toward stale weather reuse; I enforced exact-date-only weather reuse and clear failure messaging when same-date weather is unavailable.
• NOMADS requests hanging in Python: curl worked but requests could hang; I switched the downloader to sessions, streaming, explicit headers, and better timeouts so live weather fetch completes reliably.
• Missing-region weather crashes: src/weather/separateWeatherByRegion.py could fail with raw KeyError; I added clearer region-missing validation.
• No upfront runtime validation: users only discovered missing keys/tools mid-run; I added preflight checks in src/CarbonCastE2E.py for EIA_API_KEY, wgrib2, weather availability, and output directories.
• First-tier config path failure: realtime forecasting could not find firstTierConfig.json; I fixed first-tier config and model path resolution to work from the repo root.
• First-tier output directory failure: forecast CSV writes failed because fuel_forecast/ did not exist; I made first-tier create realtime output directories before writing.
• First-tier model output shape bug: some regions like PJM returned prediction arrays in a different shape, causing reshape crashes; I normalized model outputs in src/firstTierForecasts.py with safe flattening.
• Second-tier config/model path failure: realtime CI forecasting could not find secondTierConfig.json or saved models reliably; I fixed second-tier config/model path resolution in src/secondTierForecasts.py.
• Second-tier saved-model path bug: lifecycle/direct model roots were being concatenated incorrectly; I fixed the saved-model base path so lifecycle/<REGION> and direct/<REGION> resolve correctly.
• Realtime forecast CSV timestamp mismatch: second-tier loading assumed one timestamp schema while realtime files used another; I added timestamp normalization so files with UTC time also load correctly.
• Outdated Keras imports: older import paths in src/secondTierForecasts.py could break on modern environments; I updated them to current keras.layers imports.

Validation

• successful end-to-end realtime run for AECI on 2026-03-21
• successful end-to-end realtime run for AECI on 2026-03-22
• successful end-to-end realtime run for PJM on 2026-03-22