Skip to content

Reduce per-turn latency (currently ~5-11s) #6

Description

@vgpastor

Problem / Problema

EN: Each conversation turn currently takes ~5–11s end-to-end (perceptible on
WhatsApp). Most of it is model + tool round-trips. We already send a "typing…"
indicator, but the wall-clock is still high.

ES: Cada turno de conversación tarda ~5–11s de punta a punta (perceptible en
WhatsApp). La mayor parte es el modelo + los round-trips de tools. Ya enviamos el
indicador de "escribiendo…", pero el tiempo total sigue siendo alto.

Ideas to explore / Ideas a explorar

  • Measure where the time goes (add per-phase timings to the structured logs in
    src/infrastructure/observability/).
  • Reduce tool round-trips (batch/parallelize independent tool calls where safe).
  • Consider streaming partial responses where the channel supports it.
  • Cache stable lookups within a turn (e.g. emergency by slug).

Acceptance / Aceptación

  • A short write-up of where the latency is spent, backed by real timings.
  • At least one measurable improvement to median turn time.

Good starting point: src/application/conversation-service.ts and the agent
tools in src/agent/tools.ts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:agentAgent, tools and promptsenhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions