Analysis scripts for our SENSAI study. SENSAI is an LLM-powered tutoring assistant deployed on pwn.college.
Our study identified two broad behavioral categories by fitting a Mixture Hidden Markov Model (MHMM) to classified message sequences between users and SENSAI. These patterns, reactive and proactive, show two distinct modes of interaction with SENSAI.
- LLM_CODING_PIPELINE/ — Classifies learner messages into 16 behavioral categories (7 families) using OpenAI's API. Supports batch and one-shot modes.
- MHMM/ — Clusters learner interaction sequences using mixture of HMMs, then tests cluster-completion associations with within-learner GLMMs.
- REFLECTIONS/ — Analyzes end-of-semester survey data: perceived utility/usability Likert scales, TA comparisons, coded free-response.
| Component | Requirements |
|---|---|
| LLM_CODING_PIPELINE | Python 3.10+, openai pandas pydantic tqdm openpyxl |
| MHMM | R 4.0+, seqHMM dplyr tidyr lme4 emmeans |
| REFLECTIONS | Python 3.10+, numpy scipy |