AI Tutor Usage in Tennessee Schools: A Study’s Findings
In a two-year randomized study, students had access to an artificial intelligence (AI) tutor, named Khanmigo, with a remarkable 96% participation rate at least once. However, the median student utilized the AI tutor only one-third of the days that it was available. This paradox forms the crux of the study, titled “One Click Away,” conducted by economists Philip Oreopoulos and Nina Low under the National Bureau of Economic Research, which explored the usage of Khanmigo in 18 middle schools across Tennessee.
The students participated in daily math intervention sessions via Khan Academy, where Khanmigo was designed to offer coaching rather than direct answers. Compared to their peers receiving standard remedial instruction, students in the intervention sessions exhibited modest improvements in mathematical skills, comparable to what would be anticipated from regular Khan Academy practice devoid of AI involvement. Notably, most students turned to the AI tutor infrequently, with conversations often steering away from math-related topics, indicating a lack of initiative to fully leverage the tool.
Khan Academy’s Response to Low Engagement
In light of the study’s findings, Khan Academy’s CEO, Sal Khan, acknowledged in a blog post that the limited utilization of Khanmigo aligned with the organization’s previous assessments. In a conversation with Chalkbeat, he emphasized the crucial role of student motivation in engagement with the AI tutor.
This revelation is not surprising to educators, especially given the significant investments made under the assumption that a highly capable AI tutor would naturally attract student interaction. For instance, around 4,350 licenses were procured by Maryland for a two-year pilot program, while Indiana initiated a similar program across 112 schools.
Revising Engagement Strategies
In response, Khan Academy has recalibrated its approach. In a July blog entry, Khan suggested that the AI tutor must be integrated into the learning content itself, rather than existing as a standalone option. The development team subsequently embedded Khanmigo directly into practice problems, transforming the way students interacted with the tutor.
However, an important nuance emerges: not every student necessarily benefits from an AI tutor. Many can excel in subjects like calculus or essay writing without any AI assistance. Teachers have valid reasons to maintain a distance from AI in their classrooms.
For AI to effectively augment students’ learning, it cannot remain an optional resource. When students have the choice, they often opt for the chatbot that provides straightforward answers rather than engaging in thoughtful dialogue with a tutor. This trend echoed in both the Tennessee study and during my personal experience in classroom settings, where mandatory engagement with AI tools resulted in higher efforts from students.
Innovative Assessment Approaches
The educational field may benefit from redefining assessment protocols to include the quality of interactions with AI as a measuring tool. In a concurrent study entitled “From Tutor to Test,” with contributions from Aimée Skidmore, we assessed 12th-grade students in a Media and Communications course at Collège du Léman International School in Switzerland. Over four weeks, students analyzed their AI interactions and collaboratively developed standards for effective communication with AI, which shaped their subsequent group projects.
Following the exercise, 18 of the 21 participating students reported a shift in their attitudes towards AI, and confidence in generating high-quality responses increased significantly. However, some students exhibited overconfidence without appropriate judgement, prompting further investigation into the implications of AI usage in educational contexts.
Personal Insights from AI Engagement
Building on our earlier research, I conducted my own series of experiments as part of my master’s degree in fine arts, testing two methods of AI interaction: one as a traditional Socratic tutor and another through role-play scenarios. My objective was to evaluate their impact on critical thinking, writing skills, and creativity. The AI as the Socratic tutor proved dull, leading me to prefer unmediated approaches or the shortcuts offered by more compliant chatbots.
Conversely, role-playing with AI generated higher engagement levels, primarily due to the accountability imposed by my instructor’s evaluation of our interactions. The knowledge that my tutor would assess my contributions heightened my desire to think critically.
Real-World Classroom Applications
Based on the findings from both research studies, I developed a platform called AI Friction Labs to explore the educational benefits of role-play scenarios, which we called Friction Bots. In observational trials, students in various classrooms engaged in simulations where they needed to articulate complex concepts to an AI character, resulting in improved understanding.
For instance, in Bill Fallon’s calculus class at Cheshire Academy, students played a role in teaching a fictional peer grappling with tangent lines, requiring them to communicate clearly and effectively. This exercise yielded positive outcomes, as students who repeated the simulations achieved higher scores, indicating a deeper grasp of the material.
Challenges and Limitations
While many students found value in these interactive tasks, some expressed frustration, citing feelings of being interrogated or overwhelmed. Such sentiments highlight the need for better tuning and support strategies from educators effectively implementing AI in their curricula.
Frustration is an expected component of learning, as emphasized by Fallon, who remarked on the necessity of developing resilience in overcoming challenging interactions with AI, mirroring real-life scenarios.
The Path Forward
Sal Khan’s observations about motivation hold significant weight in educational discourse. While educators can dismiss the introduction of AI as irrelevant to their classes, those wishing to incorporate it must recognize that its optional use does not encourage meaningful interaction. Mandating AI’s utilization alone is insufficient; for students to genuinely engage, they must understand the expectations and be subject to assessment from credible evaluators.
Traditional views on self-directed learning—a model where students independently navigate their interactions with AI—are evolving. This newer paradigm reintegrates the teacher’s role, facilitating a systematic, evaluative approach to student dialogues with AI.
AI can enhance assessments by simulating realistic scenarios, compelling students to utilize essential skills, with educators able to monitor and evaluate performance effectively. Overall, the objective remains to leverage AI not merely as a distant tool, but as an integral aspect of the learning experience that fosters deeper engagement in education.
Discussion about this post