Research · 2026 / Agents & evaluation
Comparing human and AI evaluations
Human–machine alignment in LLM evaluation
I have worked on graph-based multi-agent systems for question generation and assessment. This raises a broader evaluation problem: a model’s judgment may not reflect the criteria a human evaluator intends.
Our 2026 study examines pretest-question evaluation through a 2 × 2 design that varies rubric operationalization and evaluation mode. The study finds systematic disagreements between human and machine judgments.
Rubric revision had a larger effect on alignment than rationale-first evaluation, with the two interventions providing complementary benefits. The work informs how we define and test evaluation criteria for LLM-based systems.
Research approach
- Make evaluation criteria explicit.
- Compare human and machine judgments.
- Investigate rubric design and evaluation mode separately.