TestCompass：不生成测试的 AI 如何引导单元测试学习？一项对比研究

View PDF HTML (experimental)

Abstract:Recent advances in generative AI have enabled assistants to generate increasingly sophisticated executable unit tests. However, novices may accept these outputs without independently identifying what to test, evaluating test-suite completeness, or verifying correctness, potentially limiting opportunities to develop essential testing skills. Although prior work has explored AI-assisted test generation and feedback-oriented programming tutors, little is known about whether AI can support novices' testing practices without generating tests for them. To investigate this, we introduce TestCompass, a generative AI tutor that provides targeted feedback on weaknesses in students' test suites without generating executable tests. In a classroom study with 18 students, we compared TestCompass with a general AI assistant across two Java unit-testing tasks, examining students' testing behaviors, test-suite quality, and subjective experiences. We found that participants using the general-purpose AI assistant produced more tests, whereas TestCompass participants spent more time developing and verifying tests themselves. However, most test-suite quality measures were comparable between conditions, and TestCompass participants achieved similar requirement coverage with significantly fewer test methods ($p=0.012$) in one task. Neither group reported short-term improvements in perceived testing knowledge or confidence. These findings suggest that feedback-oriented AI can support meaningful testing outcomes while preserving opportunities for hands-on testing practice, offering a complementary approach to AI-assisted testing education.

Submission history

From: Zihan Fang [view email]
[v1] Mon, 26 Oct 2026 19:18:29 UTC (2,460 KB)