Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Aug 25, 2025

The WebIntelligence system was overly complex and brittle, making it difficult for users to understand and maintain. This PR dramatically simplifies the system to be "actually usable by a normal human being" as requested.

What was wrong

The original WebIntelligence plugin used a complex multi-stage task pipeline:

  1. ObserveAsync → created PlanSearch tasks
  2. ExecutePlanAsync → created multiple ExecuteSearch tasks
  3. ExecuteSearchAsync → finally performed searches and added to Brain
  4. ResearchStore → complex intermediate tracking system
  5. Multiple payload types, JSON serialization, correlation keys, and staged messages

This over-engineering made the system brittle and hard to follow.

What this fixes

Simplified Architecture:

  • Direct Processing: ObserveAsync now processes unfamiliar topics immediately
  • No Intermediate Tasks: Eliminated the complex PlanSearch→ExecuteSearch pipeline
  • Uses Existing Structures: Leverages LLMSystem.Bot.Brain.Memories and ChatSession.NewTopics.Unfamiliar_Topics directly
  • Built-in Duplicate Prevention: Checks existing Brain memories by topic name before searching

Code Reduction:

  • WebIntelligencePlugin.cs: 230 → 177 lines (-23%)
  • ResearchStore.cs: Completely removed (-83 lines)
  • Total: ~136 lines of complex code eliminated

How it works now

// Simple, direct flow:
1. Check if session has unfamiliar topics
2. For each topic: check if already exists in Brain (prevent duplicates)
3. Skip low-priority topics if budget is low  
4. Perform web search → merge results → add directly to Brain.Memories
5. Done - no complex state tracking needed

The system now automatically processes unfamiliar topics from previous chat sessions, ensures no duplicates in the Brain, and adds knowledge seamlessly in the background. It "just works" without requiring deep understanding of complex task orchestration.

Testing

The simplified logic has been verified to:

  • ✅ Prevent duplicate topics using existing Brain.Memories
  • ✅ Respect daily search budgets and urgency filtering
  • ✅ Process topics directly without intermediate storage
  • ✅ Maintain all original functionality with much simpler code

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@SerialKicked SerialKicked deleted the copilot/fix-f65f9555-67ea-4f0a-80a0-989602542b9a branch August 25, 2025 17:21
Copilot AI restored the copilot/fix-f65f9555-67ea-4f0a-80a0-989602542b9a branch August 25, 2025 17:21
Copilot AI changed the title [WIP] The Agent part, especially WebIntelligence, is way too brittle and over engineered. Simplify that system to be actually usable by a normal human being. Agent should be able to execute arbitrary tasks in the background using the LLM and other function... Simplify WebIntelligence: Remove over-engineered task pipeline and use existing Brain/NewTopics structures Aug 25, 2025
Copilot AI requested a review from SerialKicked August 25, 2025 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants