visual-emotions

Camera-based visual affect/focus metrics service. Captures webcam frames, extracts MediaPipe face+pose signals, computes normalized scores (mood, focus, frustration, confusion, arousal, drowsiness), prints live bar charts to stdout, and streams metrics over gRPC a few times per second.

Features

• MediaPipe Face & Pose Tasks for blendshapes + head pose + hand/face proximity.
• Devflow-style state scores (DISTRACTED, RESTING, CONFUSED, FRUSTRATED, DESPERATE, WONDERING, DEEP THINKING, INSPECTING, LEAN BACK, NEUTRAL), smoothed and exposed in gRPC payload and console.
• gRPC publisher (see proto/metrics.proto; stub in visual_emotions.protos).
• Console bar chart focused on state scores for visibility when running in a container/TTY.
• Configurable source ID, gRPC target, cadence, and thresholds via CLI or ServiceConfig.

Quickstart

1. Ensure MediaPipe task files are available:
   • models/face_landmarker.task
   • models/pose_landmarker.task
2. Install (Poetry): poetry install
3. Run:

   poetry run visual-emotions --camera 0 --grpc-target localhost:50051

   Exit with Ctrl+C.

CLI flags of interest:

• --send-interval (default 0.5s), --log-interval (default 0.5s).
• --calibration-frames (default 30) to set baseline pitch/yaw/roll, shoulder width, brow spacing, and blink BPM.
• --max-frames for batch/smoke runs.

gRPC schema

• Defined in proto/metrics.proto; generated runtime stubs live under src via grpc_tools.protoc.
• Package: camera. Service: CameraMetricsService.PublishMetric(Metrics) -> google.protobuf.Empty. Go package option is set for downstream consumers.
• Message Metrics fields: timestamp_ms, signal_confidence, and per-state fields (distracted, resting, confused, frustrated, desperate, wondering, deep_thinking, inspecting, lean_back, neutral, happy).
• Regenerate with: poetry run python -m grpc_tools.protoc --python_out=src --grpc_python_out=src --proto_path=proto proto/metrics.proto.

Runtime behavior

• Calibration: first face-present frames (with 5s grace on re-entry) average head pose, shoulder width (with min/max), brow spacing, and blink BPM to form baselines.
• Per frame: detect face/pose → build signals → compute state scores → print bar chart → send gRPC batch (single sample) on cadence.
• Confidence decays when face/pose missing; metrics decay toward neutral rather than zeroing.
• Dry mode: pass --no-send to skip gRPC publishing while keeping console bars active.
• Jitter control: state scores are smoothed with asymmetric EWMA (rise 0.35, fall 0.12) plus history window smoothing to keep charts stable.

Containerization

• Service is CLI-driven; wrap in your base image with dependencies installed and models mounted at /app/models/. Example run command:

  visual-emotions --camera 0 --grpc-target your-host:50051 --source dev-laptop