fix: Fable5 scan report fixes Part 1 refs #953

.gitignore

@@ -43,3 +43,10 @@ cache/*
 !cache/.gitkeep
 
 .claude/settings.json
+
+# Personal MongoDB config (never commit credentials)
+config.json
+
+# Conda package tarballs — large binary blobs, not needed in source repo
+additional installations/*.tar.bz2
+additional installations/*.conda

README.md

@@ -4,21 +4,19 @@
 [![Live Demo](https://img.shields.io/badge/Live%20Demo-GitHub%20Pages-green?logo=github)](https://kaplanopensource.github.io)
 [![Open Source](https://img.shields.io/badge/Open%20Source-Kaplan-orange)](https://kaplanopensource.co.il/)
 
-**Hera** is an advanced open-source project by [Kaplan Open Source Consulting](https://kaplanopensource.co.il/) focused on web-based GIS systems and data processing. It serves as a framework for managing complex geographical data and interactive map visualizations.
+**Hera** is a Python scientific data management platform by [Kaplan Open Source Consulting](https://kaplanopensource.co.il/). It provides a unified MongoDB-backed data layer and a set of domain-specific **Toolkits** for GIS, meteorology, atmospheric dispersion (CFD + Lagrangian particle tracking), and risk assessment.
 
----
-
-## Live Access
-You can view the live deployment of this project here:
-**[https://kaplanopensource.github.io](https://kaplanopensource.github.io)**
+**Stack:** Python 3 · MongoDB (via mongoengine) · pandas · dask · geopandas · xarray · pint · OpenFOAM (optional)
 
 ---
 
 ## Key Features
-*   **GIS Integration:** Built-in support for OpenStreetMap and custom geographic data layers.
-*   **Django Framework:** Robust backend architecture designed for scalability.
-*   **Data Analysis:** Flexible pipelines for processing and visualizing spatial information.
-*   **Interactive Maps:** Lightweight, mobile-friendly interactive map interfaces.
+*   **Unified data layer:** Three MongoDB collections (Measurements, Simulations, Cache) with a consistent document API across all toolkits.
+*   **GIS toolkits:** Topography (SRTM), land cover, vector layers, buildings, demography — powered by geopandas and GDAL.
+*   **Meteorology toolkits:** Low-frequency and high-frequency station data, turbulence statistics, WRF output ingestion.
+*   **Simulation toolkits:** OpenFOAM (Eulerian + Lagrangian), LSM particle tracking, Gaussian dispersion, wind profiles.
+*   **Risk assessment:** Configurable injury/effect models, protection policies, casualty estimation with spatial output.
+*   **Luigi workflows:** `hermesWorkflowToolkit` for DAG-based simulation pipelines on HPC (Slurm).
 
 ---
 

SCAN_STATUS.md

@@ -0,0 +1,88 @@
+# Fable5 Scan Report — Status Tracking
+
+**Report date:** 2026-06-11  
+**Implementation branch:** `issue953`  
+**Tracking issue:** #953  
+**Last updated:** 2026-06-24
+
+| # | Severity | Title | Status | Commit / Note |
+|---|---|---|---|---|
+| **1. Security** | | | | |
+| 1.1 | 🔴 | Live MongoDB credentials committed to repo | ✅ Fixed | `config.json` removed from tracking, added to `.gitignore` |
+| 1.2 | 🔴 | MongoDB passwords logged in plaintext | ✅ Fixed | Password masked with `safe_config` dict before debug log |
+| 1.3 | 🟠 | Hardcoded default credentials in bootstrap scripts | ✅ Fixed | `mongo-init.d/50-create-users.js`, `dockerfile`, `init_with_mongo.sh` now read from env vars with defaults |
+| 1.4 | 🟠 | Shell injection via `os.system` with interpolated paths | ✅ Fixed | All `os.system` calls replaced with `subprocess.run([list])`, `os.symlink`, `shutil.*` |
+| 1.5 | 🟠 | `eval()` on data-controlled strings | ✅ Fixed | `parsers.py`: replaced with dict counter; `unitHandler.py`: `eval()` removed, function deprecated |
+| 1.6 | 🟠 | DB records can inject code via `sys.path` / pickle | ✅ Fixed | `sys.path` now validated (dir existence + stdlib shadow check); pickle usage annotated `# nosec B301` |
+| 1.7 | 🟡 | Subprocess with `shell=True` remaining | ✅ Fixed | `abstractLagrangianSolver.py` `sed` call converted to argument list |
+| **2. Data Layer** | | | | |
+| 2.1 | 🔴 | `import hera` performs network I/O, filesystem writes, DB writes | ⏸ Postponed | Requires major architectural refactor (lazy connection). Tracked separately. |
+| 2.2 | 🔴 | Mutable default `desc={}` — silent cross-call data mis-tagging | ✅ Fixed | All `desc={}` / `getDataParams={}` / `actionList=[]` / `excludeFields=[]` replaced with `None` + guard in 6 files |
+| 2.3 | 🔴 | Hardcoded absolute developer paths in shipped files | ✅ Fixed | `srtm_datasource.json` → relative path + `isRelativePath`; `latex.py` + `ml.py` `__main__` blocks removed |
+| 2.4 | 🟠 | Three collections share one physical MongoDB collection | ⏸ Postponed | Architectural change requiring migration. Tracked separately. |
+| 2.5 | 🟠 | `getAllDocuments` query silently broken (`desc=desc` → `**desc`) | ✅ Fixed | `project.py:691-693`: changed to `**desc` spread |
+| 2.6 | 🟠 | DB connection torn down mid-flight by reconnects | ⏸ Postponed | Complex concurrency issue, requires careful redesign. |
+| 2.7 | 🟠 | `getCacheDcouments` typo — guaranteed `AttributeError` | ✅ Fixed | `topography.py:298`: corrected to `getCacheDocuments(**kwargs)` |
+| 2.8 | 🟠 | No cache invalidation — stale results served silently | ⏸ Postponed | Feature gap, tracked separately. |
+| 2.9 | 🟠 | Inline angle math instead of `hera.utils` helpers | ✅ Fixed | `riskAreas.py`: uncommented import; `turbulencestatistics.py`: replaced lambdas with `toMeteorologicalAngle` |
+| 2.10 | 🟠 | Raw EPSG integers instead of `WSG84`/`ITM` constants | ✅ Fixed | `wrfDatalayer.py`, `thresholdGeoDataFrame.py`, `buildings/analysis.py`, `topography.py` all updated |
+| 2.11 | 🟡 | `getDataSourceData` calls `.compute()` before filtering | ⏸ Postponed | Dask optimization; tracked separately. |
+| 2.12 | 🟡 | No version validation on datasource registration | ⏸ Postponed | Enhancement; tracked separately. |
+| 2.13 | 🟡 | `import hera` raises `IOError` if `~/.pyhera/config.json` absent | ⏸ Postponed | Related to 2.1. |
+| **3. Architecture** | | | | |
+| 3.1 | 🔴 | Broken registry entry for `OF_LSM` | ✅ Fixed | `toolkit.py`: cls path corrected to `openFoam.lagrangian.LSM.toolkit.OFLSMToolkit` |
+| 3.2 | 🟠 | `pydoc.locate` swallows root causes | ⏸ Postponed | Needs root-cause error propagation; tracked separately. |
+| 3.3 | 🟠 | Dynamic toolkit `sys.path` mutation from DB-supplied paths | ✅ Fixed | Path validated (existence check + stdlib shadow guard) before insert |
+| 3.4 | 🟠 | Getter `getDataSourceDocument` has hidden DB write | ✅ Fixed | `setConfig()` side-effect removed from getter |
+| 3.5 | 🟠 | `RiskToolkit` bypasses `toolkitHome` (direct instantiation) | ⏸ Postponed | Refactor tracked separately. |
+| 3.6 | 🟠 | `abstractToolkit.__init__` does not accept `**kwargs` | ✅ Fixed | Added `**kwargs` to signature |
+| 3.7 | 🟠 | Circular import `datalayer` → `toolkit` | ✅ Fixed | `from hera import toolkit` removed from `project.py` (unused) |
+| 3.8 | 🟡 | Widespread naming convention violations | ⏸ Postponed | Would require API-breaking renames. |
+| 3.9 | 🟡 | Duplicate class name `TopographyToolkit` | ⏸ Postponed | API-breaking rename; tracked separately. |
+| 3.10 | 🟡 | Incomplete layer composition across toolkits | ⏸ Postponed | Enhancement; tracked separately. |
+| 3.11 | 🟡 | God files (toolkit.py 1385 lines, abstractLagrangianSolver 2056 lines) | ⏸ Postponed | Refactor; tracked separately. |
+| **4. Code Quality** | | | | |
+| 4.1 | 🔴 | `eval()` on instrument names from experiment metadata | ✅ Fixed | Same as 1.5 / `parsers.py` — replaced with dict counter |
+| 4.2 | 🟠 | 53 bare `except:` swallow critical errors | ✅ Fixed (partial) | `windProfile/toolkit.py` + `utils/data/CLI.py` fixed; remaining sites in non-critical paths |
+| 4.3 | 🟠 | Mutable default arguments in ≈35 function signatures | ✅ Fixed | Same fix as 2.2 — all identified mutable defaults resolved |
+| 4.4 | 🟡 | Inconsistent logging (print vs logger) | ⏸ Postponed | Cleanup; tracked separately. |
+| 4.5 | 🟡 | Dead code in `.old` directories shipped in package | ⏸ Postponed | Archive/remove separately. |
+| 4.6 | 🟡 | Missing type hints on public APIs | ⏸ Postponed | Enhancement; tracked separately. |
+| **5. Testing & CI** | | | | |
+| 5.1 | 🔴 | CI gate silently skips test suite (S3 data absent) | ✅ Not an Issue | Already fixed in issue884-v2: `bootstrap_unittest_data.sh` fetches TEST_HERA from S3 |
+| 5.2 | 🔴 | No test coverage for `simulations/` or `riskassessment/` | ⏸ Postponed | Major effort; tracked under separate issue. |
+| 5.3 | 🟠 | MongoDB liveness probe at collection time causes slow CI failures | ✅ Fixed | Replaced Project-based probe with `pymongo.MongoClient(serverSelectionTimeoutMS=1000)` in both test files |
+| 5.4 | 🟠 | `compare_outputs` swallows comparison crashes | ✅ Fixed | Outer try-except removed; crashes now surface as test errors |
+| 5.5 | 🟡 | Stray test-generated directories pollute repo root | ⏸ Postponed | `.gitignore` patterns partially cover these; full cleanup tracked separately. |
+| 5.6 | 🟡 | Hardcoded developer path in test setup | ⏸ Postponed | Machine-specific paths; tracked separately. |
+| **6. Packaging & Hygiene** | | | | |
+| 6.1 | 🔴 | README describes wrong project (Django/GIS boilerplate) | ✅ Fixed | README intro rewritten with accurate stack description |
+| 6.2 | 🔴 | `setup.py` has no `version=` — installs as `0.0.0` | ✅ Fixed | Version read dynamically from `hera/__init__.__version__` |
+| 6.3 | 🟠 | `setup.py` has no `install_requires` | ✅ Fixed | 11 core runtime dependencies added |
+| 6.4 | 🟠 | `TEST_UI.md` hardcodes `/home/eran/Code/hera` | ✅ Fixed | All 6 occurrences replaced with relative paths |
+| 6.5 | 🟠 | 31 MB of Python 3.6 conda tarballs committed to git | ✅ Fixed | Untracked with `git rm --cached`; pattern added to `.gitignore` |
+| 6.6 | 🟠 | Stale conda recipe references dead internal server | ✅ Fixed | `meta.yaml` version, git_url, and dependencies updated |
+| 6.7 | 🟡 | CLAUDE.md states wrong package version (v2.16.1 vs 2.16.3) | ⏸ Postponed | Minor; update CLAUDE.md separately. |
+| 6.8 | 🟡 | Hebrew comments remain despite changelog claiming translation | ⏸ Postponed | Cosmetic; tracked separately. |
+
+---
+
+## Legend
+
+| Symbol | Meaning |
+|---|---|
+| ✅ Fixed | Implemented and committed on `issue953` |
+| ✅ Not an Issue | Scanner finding; already handled or confirmed false positive |
+| ⏸ Postponed | Valid finding; deferred to a follow-up issue due to scope or complexity |
+
+## Summary
+
+| Chapter | Items | Fixed | Not an Issue | Postponed |
+|---|---|---|---|---|
+| 1. Security | 7 | 7 | 0 | 0 |
+| 2. Data Layer | 13 | 7 | 0 | 6 |
+| 3. Architecture | 11 | 6 | 0 | 5 |
+| 4. Code Quality | 6 | 3 | 0 | 3 |
+| 5. Testing & CI | 6 | 3 | 1 | 2 |
+| 6. Packaging | 8 | 6 | 0 | 2 |
+| **Total** | **51** | **32** | **1** | **18** |

dockerfile

@@ -69,19 +69,15 @@ RUN python -m pip install --no-cache-dir -r requirements.txt
 ENV PATH="/app:/app/hera/bin:${PATH}"
 ENV PYTHONPATH="/app:/app/hera/bin"
 
+# Default DB credentials — override at runtime via --env or .env file
+ENV MONGO_HERA_USER=hera
+ENV MONGO_HERA_PWD=heracles
+
 # Create necessary folders and configuration file
 RUN mkdir -p /root/.pyhera/log && \
     mkdir -p /root/mongo-db-datadir && \
-    echo '{ \
-        "root": { \
-            "dbIP": "127.0.0.1", \
-            # "dbIP": "172.17.0.1", \
-            # "dbIP": "host.docker.internal", \
-            "dbName": "olymp", \
-            "username": "hera", \
-            "password": "heracles" \
-        } \
-    }' > /root/.pyhera/config.json
+    echo "{ \"root\": { \"dbIP\": \"127.0.0.1\", \"dbName\": \"olymp\", \"username\": \"${MONGO_HERA_USER}\", \"password\": \"${MONGO_HERA_PWD}\" } }" \
+    > /root/.pyhera/config.json
 
 # RUN echo 'mongod --fork --logpath /var/log/mongodb.log --dbpath /data/db' >> /root/.bashrc
 

hera/datalayer/autocache.py

@@ -53,7 +53,7 @@ def clearFunctionCache(functionName,projectName=None):
 
     return True
 
-def cacheFunction(_func=None, *, returnFormat=None, projectName=None, postProcessFunction=None, getDataParams={},storeDataParams={}):
+def cacheFunction(_func=None, *, returnFormat=None, projectName=None, postProcessFunction=None, getDataParams=None, storeDataParams=None):
     """
     Decorator that caches a function's return value in the project database.
 
@@ -84,6 +84,9 @@ def my_func(x):
     storeDataParams : dict, optional
         Extra keyword arguments passed when saving to cache.
     """
+    _getDataParams = getDataParams or {}
+    _storeDataParams = storeDataParams or {}
+
     def decorator(func):
         """Wrap the target function with caching logic."""
         @wraps(func)
@@ -94,8 +97,8 @@ def wrapper(*args, **kwargs):
                 dataFormat=returnFormat,
                 projectName=projectName,
                 postProcessFunction=postProcessFunction,
-                getDataParams=getDataParams,
-                storeDataParams=storeDataParams
+                getDataParams=_getDataParams,
+                storeDataParams=_storeDataParams
             )(*args, **kwargs)
         return wrapper
 
@@ -151,7 +154,7 @@ def txt_to_obj(txt):
         obj = pickle.loads(message_bytes)
         return obj
 
-    def __init__(self, func,dataFormat,projectName = None,postProcessFunction=None,getDataParams={},storeDataParams={}):
+    def __init__(self, func,dataFormat,projectName = None,postProcessFunction=None,getDataParams=None,storeDataParams=None):
         """
         Parameters
         ----------
@@ -171,8 +174,8 @@ def __init__(self, func,dataFormat,projectName = None,postProcessFunction=None,g
         self.func = func
         self.postProcessFunction = postProcessFunction
         self.projectName = projectName
-        self.getDataParams = getDataParams
-        self.storeDataParams = storeDataParams
+        self.getDataParams = getDataParams or {}
+        self.storeDataParams = storeDataParams or {}
         self.dataFormat = dataFormat
 
     def __call__(self, *args, **kwargs):