added SBERT and LLM as judge - #5
Open
JamesVillanueva-Dev wants to merge 4 commits into
Open
Conversation
jjdrisco
pushed a commit
that referenced
this pull request
Jun 8, 2026
…egrate PR #5 Phase 1: Extract DeBERTaV3 into modular structure - Move data models to utils/data_models.py - Extract DeBERTaV3 backend to backends/deberta.py - Extract scoring logic to scoring/deberta_scoring.py - Extract reporting to reporting/deberta_report.py - Remove monolithic metrics.py and reporting.py Phase 2: Implement StanceAware backend from PR #5 - Add StanceAwareEmbedder to backends/stanceaware.py - Add StanceAware scoring to scoring/stanceaware_scoring.py - Add StanceAware reporting to reporting/stanceaware_report.py Phase 3: Implement LLM Judge backend from PR #5 - Add OpenAIJudge to backends/llm_judge.py - Add LLM judge scoring to scoring/llm_judge_scoring.py - Add LLM judge reporting to reporting/llm_judge_report.py Phase 4: Multi-evaluator CLI orchestration - Update cli.py to support --evaluators flag - Run evaluators independently with separate output directories - Create summary.txt with status of each evaluator Benefits: - Modular: Each evaluator independent, can add more easily - Scalable: Follows same pattern as turn_taking_length - Backward compatible: Output identical to original implementation - Flexible: CLI supports selective evaluator execution All existing tests pass with updated imports.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SBERT seems to work and LLM uses OPENAI as the judge.