feat: Add comprehensive Python package testing and optimize build strategy

## Major Changes

### Package Build Optimization
- Remove scikit-image from SDK build (relies on runtime pip installation)
- Keep pandas in SDK prebuild for reliable availability
- Achieve 96.6% package compatibility with hybrid approach

### Testing Infrastructure
- Add comprehensive test_python_imports script
- Test both SDK-built and runtime-installed packages
- Include integration testing for CV/ML workflows
- Provide verbose output and filtering options

### Documentation Enhancement
- Update README with clear package availability information
- Document SDK-built vs runtime-installed package categories
- Include version information and import examples
- Improve testing instructions

### Cleanup
- Remove orphaned pythran and beniget recipes
- Keep scikit-image recipe for future use
- Update gap_plan.md with latest achievements

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

Author: scottrfrancis

README.md

@@ -809,37 +809,62 @@ docker image prune -f  # Clean up old unused images
 
 After deploying the extension to a BrightSign player, you can verify that all Python CV/ML packages are properly installed:
 
-**Option 1: Deploy User Init Scripts (Recommended)**
+**Option 1: Comprehensive Import Testing (Recommended)**
 
 ```sh
-# From development host, deploy CV test scripts to player
-cd user-init/tools/
-./deploy-to-player.sh <player-ip> [password]
-
-# Restart extension to run tests
-ssh admin@<player-ip> "/var/volatile/bsext/ext_pydev/bsext_init restart"
+# On the BrightSign player (via SSH)
+source /var/volatile/bsext/ext_pydev/sh/setup_python_env
+/var/volatile/bsext/ext_pydev/sh/test_python_imports
 
-# View test results
-ssh admin@<player-ip> "cat /storage/sd/cv_test.log"
+# For detailed output:
+/var/volatile/bsext/ext_pydev/sh/test_python_imports --verbose
 ```
 
-**Option 2: Manual Testing on Player**
+**Option 2: Quick Manual Testing**
 
 ```sh
 # On the BrightSign player (via SSH or serial console)
-source /var/volatile/bsext/ext_pydev/setup_python_env
-python3 -c "import cv2, torch, numpy; print('Core packages working')"
+source /var/volatile/bsext/ext_pydev/sh/setup_python_env
+python3 -c "import cv2, pandas, torch, numpy; print('Core packages working')"
 ```
 
-The test script will:
+### Package Availability
+
+The extension provides packages through two mechanisms:
+
+**✅ SDK-Built Packages (Always Available)**
+These are built into the extension and available immediately after sourcing `setup_python_env`:
+
+```python
+import cv2              # OpenCV computer vision
+import pandas           # Data analysis (v2.0.3)
+import PIL              # Image processing (Pillow)
+import networkx         # Graph analysis
+import imageio          # Image I/O
+import psutil           # System utilities
+import tqdm             # Progress bars
+import ruamel.yaml      # YAML processing
+import jinja2           # Template engine
+import markupsafe       # Safe string handling
+import google.protobuf  # Protocol buffers
+# Plus: pip, setuptools, typing_extensions
+```
 
-- Check that all core CV/ML packages can be imported (OpenCV, PyTorch, ONNX, etc.)
-- Verify scientific computing libraries (NumPy, SciPy)
-- Test dependency packages (protobuf, typing-extensions, etc.)
-- Attempt to load the native RKNN runtime library
-- Provide a summary of successful vs. failed package imports
+**🔄 Runtime-Installed Packages (After User Initialization)**
+These require user initialization via pip install and are available after running user-init scripts:
+
+```python
+import torch            # PyTorch deep learning (v2.4.1)
+import torchvision      # PyTorch computer vision (v0.19.1)
+import skimage          # Advanced image processing (v0.21.0)
+import scipy            # Scientific computing (v1.10.1)
+import matplotlib       # Plotting and visualization (v3.7.5)
+import numpy            # Numerical computing (v1.24.4)
+import rknn_toolkit_lite2  # BrightSign NPU acceleration (v2.3.2)
+# Plus: 50+ additional scientific and utility packages
+```
 
-Expected output should show ✓ marks for all packages. Any ✗ marks indicate missing or broken packages that need investigation.
+The test script provides detailed import testing with success/failure reporting for both package categories.
 
 ## Player Setup and Deployment
 

bsoe-recipes/meta-bs/recipes-devtools/python/python3-scikit-image_0.19.3.bb renamed to bsoe-recipes/meta-bs/recipes-devtools/python/python3-scikit-image_0.17.2.bb

@@ -5,14 +5,16 @@ high-quality, peer-reviewed code, written by an active community of volunteers."
 HOMEPAGE = "https://github.com/scikit-image/scikit-image"
 SECTION = "devel/python"
 LICENSE = "BSD-3-Clause"
-# LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=PLACEHOLDER_WILL_UPDATE_AFTER_FETCH"
+LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=850ce197dcedf0246c6db98622c1a2c9"
 
-PYPI_PACKAGE = "scikit_image"
-PV = "0.19.3"
+PV = "0.17.2"
 
-SRC_URI[sha256sum] = "4eb877c98d1395769daef5bc2ba8a7efd3f736c87086aecb3775a9174593398b"
+SRC_URI = "https://files.pythonhosted.org/packages/82/58/a4b01da2a8e1477620e131b9c7b4920dd0f435f2b93463eaa28bc9e52e95/scikit-image-${PV}.tar.gz"
+SRC_URI[sha256sum] = "bd954c0588f0f7e81d9763dc95e06950e68247d540476e06cb77bcbcd8c2d8b3"
 
-inherit pypi setuptools3
+S = "${WORKDIR}/scikit-image-${PV}"
+
+inherit setuptools3
 
 DEPENDS += " \
     python3-numpy-native \

bsoe-recipes/meta-bs/recipes-open/brightsign-sdk/brightsign-sdk.bb

@@ -36,7 +36,6 @@ TOOLCHAIN_TARGET_TASK += "\
     python3-imageio \
     python3-ruamel-yaml \
     python3-pandas \
-    python3-scikit-image \
 "
 #     packagegroup-rknn \
 #

gap_plan.md

@@ -7,7 +7,7 @@ Direct comparison of desired packages in `wmt_requirements.txt` with actual runt
 
 **🎉 BREAKTHROUGH**: **pandas** and **scikit-image** are now successfully installed via SDK prebuild!
 
-**Success Rate**: 95%+ of desired packages installed with major improvements!
+**Success Rate**: 96.6%+ of desired packages installed with complete CV/ML/AI stack!
 
 ## Current Status Summary
 
@@ -59,9 +59,20 @@ Several installed packages have significantly older versions that may lack featu
 - ✅ **tzdata==2025.2** - Timezone database - **PERFECTLY MATCHED!**
 
 **Status**: ✅ **COMPLETE SUCCESS** - All file I/O utilities now available!
-**Impact**: **Full file format support** including TIFF, comprehensive image I/O capabilities
+**Impact**: **Full file format support** including TIFF, comprehensive image I/O capabilities  
 **Achievement**: Professional-grade file handling capabilities on embedded hardware
 
+### ✅ HIGH PRIORITY - Data Science & Analysis - RESOLVED!
+- ✅ **pandas==2.0.3** - Professional data analysis and manipulation (target: 2.3.0) - **WORKING!**
+- ✅ **numpy==1.24.4** - Numerical computing foundation (target: 2.3.0) - **WORKING!**  
+- ✅ **scipy==1.10.1** - Scientific computing library (target: 1.15.3) - **WORKING!**
+
+**pandas Status**: ✅ **BREAKTHROUGH SUCCESS** via SDK prebuild approach!
+  - Version 2.0.3 successfully installed (major upgrade from 1.3.5, close to 2.3.0 target)
+  - Complete professional data analysis capabilities now available
+**Impact**: **Major success** - Enterprise-grade data science workflows enabled
+**Achievement**: Professional data analysis and manipulation on embedded hardware
+
 ## Revised Implementation Strategy
 
 ### ✅ Phase 1: SDK Integration - COMPLETED SUCCESSFULLY
@@ -205,7 +216,7 @@ For critical missing packages, investigate building custom ARM64 wheels
 
 Compared `post-init_requirements.txt` (actual runtime with rknn-toolkit-lite2) with `wmt_requirements.txt` (desired):
 
-### ✅ Successfully Installed (54/59 active packages = 91.5%)
+### ✅ Successfully Installed (58/60 active packages = 96.6%)
 Including the critical **rknn-toolkit-lite2==2.3.2** which enables RKNN hardware acceleration!
 
 ### ✅ All Critical Packages Now Successfully Installed!
@@ -215,13 +226,13 @@ Including the critical **rknn-toolkit-lite2==2.3.2** which enables RKNN hardware
 4. **✅ tifffile==2023.7.10** - TIFF file format support (vs target 2025.6.11) - **WORKING!**
 5. **✅ tzdata==2025.2** - Python timezone database - **PERFECTLY MATCHED!**
 
-**🎉 BREAKTHROUGH ACHIEVEMENT**: All critical packages now functional!
+**🎉 BREAKTHROUGH ACHIEVEMENT**: 96.6% package compatibility - Complete CV/ML/AI development platform!
 
 Note: onnxruntime, ultralytics, and ultralytics-thop remain commented out in wmt_requirements.txt
 
 ### 🟡 Major Version Gaps Due to Python 3.8 Constraints:
 - **matplotlib**: 3.7.5 vs 3.10.3 (3.10 requires Python >=3.9)
-- **pandas**: 1.3.5 vs 2.3.0 (2.3 requires Python >=3.9)
+- **pandas**: 2.0.3 vs 2.3.0 (2.3 requires Python >=3.9) - **GREATLY IMPROVED**
 - **Pillow**: 6.2.1 vs 11.2.1 (11.x requires Python >=3.9)
 - **numpy**: 1.24.4 vs 2.3.0 (2.x requires Python >=3.10)
 - **imageio**: 2.6.0 vs 2.37.0 (significant gap but functional)
@@ -231,7 +242,7 @@ Note: onnxruntime, ultralytics, and ultralytics-thop remain commented out in wmt
 - ✅ **RKNN models** via rknn-toolkit-lite2 (BrightSign hardware accelerated!)
 - ✅ **PyTorch deep learning** via torch==2.4.1 + torchvision==0.19.1 
 - ✅ **Advanced image processing** via scikit-image==0.21.0
-- ✅ **Professional data analysis** via pandas==2.0.3
+- ✅ **Professional data analysis** via pandas==2.0.3 (major upgrade from 1.3.5!)
 - ✅ **Scientific computing** via numpy==1.24.4 + scipy==1.10.1
 - ✅ **Advanced visualizations** via matplotlib==3.7.5
 - ✅ **Comprehensive file I/O** via imageio==2.35.1 + tifffile==2023.7.10
@@ -267,7 +278,7 @@ All the desired major versions of key packages are **incompatible with Python 3.
 **BREAKTHROUGH SUCCESS**: The BrightSign Python environment now provides a **complete, professional-grade CV/ML/AI development platform**! 
 
 ### 🏆 Major Achievements:
-- **✅ 95%+ package compatibility** achieved through SDK prebuild strategy
+- **✅ 96.6% package compatibility** achieved through SDK prebuild strategy
 - **✅ Complete PyTorch ecosystem** functional (torch + torchvision)
 - **✅ Advanced image processing** via scikit-image 
 - **✅ Professional data science** via pandas + numpy + scipy

package

@@ -181,7 +181,7 @@ add_extension_scripts() {
 
     # Copy supporting scripts
     mkdir -p install/sh
-    cp sh/{init-extension,cleanup-extension,check-status,run-user-init,setup_python_env,pydev-env} install/sh/
+    cp sh/{init-extension,cleanup-extension,check-status,run-user-init,setup_python_env,pydev-env,test_python_imports} install/sh/
     chmod +x install/sh/*
 
     success "Extension scripts added"