For two full academic years, students in a closely watched randomized study had a sophisticated artificial intelligence tutor sitting just one click away. Of the hundreds of participants, a staggering 96% tried
the tool at least once. Yet, despite this overwhelming initial curiosity, the median student ended up utilizing the platform—an AI assistant named Khanmigo—on only about a third of the days they had access to it.
That finding forms the core of "One Click Away," a National Bureau of Economic Research (NBER) working paper authored by economists Philip Oreopoulos and Nina Low. The researchers tracked the deployment of Khan Academy’s Khanmigo across 18 middle schools in Tennessee. Throughout the study, students used Khan Academy during their daily math intervention sessions, with Khanmigo specifically configured to act as a Socratic coach rather than a shortcut dispenser that simply handed over answers.
When researchers compared these students to peers receiving traditional remedial instruction, the AI-assisted group made only modest math gains. These improvements were roughly on par with what educators would typically expect from standard Khan Academy practice without any artificial intelligence intervention. Most students engaged with the AI tutor infrequently, and their text conversations with the bot rarely touched upon mathematics. In short, when left to their own devices, students made very little effort to leverage the technology to their academic advantage.
In a subsequent blog post, Khan Academy CEO Sal Khan walked through the study’s findings, noting that the low utilization rate of Khanmigo aligned with what his organization had been observing and admitting publicly. In an interview with education publication Chalkbeat, he emphasized that the research vividly demonstrated "how much motivation matters."
Classroom teachers are unlikely to be surprised by that conclusion, even though billions of dollars and countless hours of development have been spent on the opposite foundational assumption: the premise that if developers build a patient, highly capable AI tutor and place it directly in front of students, they will naturally want to use it. The tech industry has heavily backed this thesis. Microsoft, for instance, donated enough Azure cloud infrastructure to make Khanmigo’s teacher tools entirely free for every educator across the United States. Meanwhile, state departments of education have purchased bulk licenses in attempts to modernize classrooms. Maryland acquired 4,350 licenses for a two-year pilot program, while Indiana launched a broad initiative spanning 112 schools.
How Khan Academy Adjusted
Faced with the reality of low voluntary engagement, Khan Academy has already begun changing course. In a July blog post, Sal Khan wrote candidly that an AI tutor could not simply sit passively next to the educational content; it had to be woven directly into the fabric of the assignments. The development team realized they had to "make productive struggle harder to sidestep." Earlier this year, the organization integrated Khanmigo directly into its digital practice problems, embedding the AI experience so that students encounter it automatically rather than having to deliberately open a chat window or ask for help.
That architectural fix makes logical sense for Khan Academy’s product ecosystem, but it carries a risk of misinterpretation across the broader educational landscape. Observers can easily misread the trend as proof that every student desperately needs an AI tutor. They do not. Generations of students have successfully learned complex calculus, composed powerful analytical essays, and developed critical thinking skills entirely without artificial intelligence. Furthermore, plenty of professional educators will reasonably choose to keep the technology out of their classrooms altogether.
If a school system genuinely wants artificial intelligence to support student thinking and cognitive development, relying on voluntary adoption simply does not work. When AI assistance remains optional, students rarely choose the rigorous tutor that asks probing, Socratic questions. Instead, they lean toward the compliant chatbot in the next browser tab that bypasses the struggle and simply provides the final answer.
Educators who have experimented with AI in high school and college classrooms have observed this exact pattern for years. When AI study tools are introduced as an optional resource, students consistently defer engagement. However, when educators require students to utilize specific AI tools—while explicitly monitoring the conversation logs and providing structured feedback—meaningful engagement begins to take root.
Even mandatory participation, however, only solves the baseline participation problem; it does not automatically solve the effort problem. A student can be compelled to open a chat window and talk to an artificial intelligence tutor while still contributing zero genuine intellectual effort. Kristen DiCerbo, Khan Academy’s chief learning officer, noted during an EdTech Insiders panel that her team observed "more passive kinds of interaction than we would like." A built-in tutor will successfully initiate more conversations, but it fundamentally cannot force a student to care about the quality of their work.
Requirement vs. Assessment
Many experts argue that the education sector is missing a crucial intermediary step: treating the conversational transcript itself as the primary assessment.
That exact methodology formed the design of the research detailed in "From Tutor to Test," a book chapter co-authored by Aimée Skidmore and published in the Elsevier volume GenAI in Higher Education. The study followed 21 twelfth-grade students enrolled in a Media and Communications course at the Collège du Léman International School in Switzerland. Over a four-week period, students examined side-by-side transcripts of strong and weak interactions with artificial intelligence. Alongside their teacher, they collaborated to build a comprehensive rubric defining what effective, critical AI interaction actually looks like.
Following this preparation, the students used ChatGPT to brainstorm ideas for a media analysis project and subsequently turned in their complete chat transcripts. These logs were evaluated using the newly established rubric. Students were graded not on the final essay alone, but on how precisely they formulated their initial prompts, how critically they evaluated the AI’s suggestions, and whether they managed to maintain intellectual ownership of their own thinking.
In post-project surveys, 18 of the 21 students reported that their fundamental approach to artificial intelligence had shifted. Ten students stated that they now thought much more deliberately about their own input and how to build upon it later in an ongoing conversation. Furthermore, the number of students who felt completely confident in extracting high-quality responses from the AI rose from just two to eleven.
While rising student confidence can occasionally serve as a warning sign of over-reliance, individual tracking revealed nuanced psychological shifts. Five students experienced major boosts in confidence while simultaneously developing a sharper awareness of the inherent risks of artificial intelligence. Four students actually grew less confident in their ability to accurately judge the AI’s output, a shift researchers interpreted as a healthy recalibration of skepticism.
Only two students displayed the problematic pattern researchers worried about: massive confidence gains paired with a dismissive attitude toward independent critical thought. One student, who had never used artificial intelligence prior to the course, insightfully noted that students "need to know how to approach AI to not lose their capacity to think."
Although it was a small-scale pilot study lacking a strict control group—with data relying heavily on self-reporting—the underlying pedagogical logic remains straightforward. When the live conversation with the machine serves as the test, students understand precisely what they are required to demonstrate. They are motivated to invest genuine effort because they know an authority figure whose judgment they respect will thoroughly review the results.
Firsthand Experience and Classroom Trials
The implications of these dynamics extend beyond high school media classes. Instructors experimenting with AI in higher education have found similar psychological shifts when grading the interactive process rather than just the final product. Using AI concurrently as a conventional Socratic tutor and as a role-play partner—where the machine simulates a difficult persona that must be negotiated with or taught—highlights a stark contrast in student motivation.
While a standard Socratic tutor often feels monotonous and easy to abandon for faster shortcuts, role-playing scenarios create a heightened sense of stakes. Knowing that an instructor will evaluate the performance recorded in the chat transcript changes the psychological equation. The artificial intelligence engine remains identical in both contexts, but the student’s awareness of external accountability forces a higher level of critical reasoning.
To test these hypotheses further, educators have begun deploying specialized role-play platforms in traditional academic settings. In a calculus class at Cheshire Academy in Connecticut, 21 students participated in a simulation where they assumed the role of the instructor. The AI engine played "Devon," a fictional classmate who genuinely did not understand tangent lines and refused to pretend otherwise. When students leaned lazily on jargon such as "instantaneous rate of change," Devon grew even more confused. To succeed in the simulation, students had to master the mathematical concept well enough to explain it clearly in plain language.
Following the exercise, transcripts were evaluated by both an automated algorithm and the teacher, focusing on student persistence, clarity, and adherence to the teaching persona. Five students voluntarily chose to retake the role-play simulation after reviewing their initial performance. All five scored significantly higher on their second attempt, improving by an average of about 34 points. Notably, they returned not because they were forced to, but because the transcript and AI-generated evaluation clearly exposed where their logical explanations had broken down.
Similar outcomes emerged in science classrooms at San Domenico School in California, where an AI program simulated a seasoned ecologist pressing students with relentless follow-up questions. While one student remarked that the rigorous questioning felt close to "interrogation," another observed that it "forced me to go beyond memorizing terms and actually understand the mechanisms behind them."
Naturally, not all students enjoy these frictions. Some complain that repetitive, probing questions feel frustrating or even antagonistic. Yet educators point out that navigating productive frustration is a vital cognitive and emotional skill. As one participating calculus teacher noted, real life is filled with frustrating interactions; learning how to work through them productively represents an entirely new level of thought and education. Students rarely express deep affection for traditional written examinations either, but educators do not take that as evidence that tests measure nothing.
Where This Leaves Us
The broader debate ultimately circles back to the central role of human motivation and teacher oversight. Khan Academy’s leadership is correct in asserting that motivation is paramount. Schools retain the right to decide that artificial intelligence has no place within their walls, and that remains a legitimate pedagogical choice. However, in environments where educators do wish to leverage AI to support critical thinking, making the technology optional fundamentally undermines the goal. Students will naturally default to the path of least resistance, choosing the tool that performs the cognitive labor for them.
At the same time, simply mandating the use of AI tools is insufficient. Students exert effort when they understand the criteria they must meet and recognize that a respected human evaluator will examine their work. This principle underpinned effective assessment long before the advent of generative AI.
Much of the early commercial vision surrounding artificial intelligence tutors rested heavily on the ideal of isolated, self-directed learning—a motivated student sitting alone with a patient digital machine. The emerging alternative model places the human teacher firmly back at the center of the instructional ecosystem. Automated systems can generate chat transcripts revealing how a student reasoned in real time, where they stumbled, and how they recovered. AI engines can draft initial evaluations based on predefined criteria, but the human teacher makes the final judgment, adding notes and deciding what feedback goes back to the student.
As educators gain experience sifting through thousands of pages of student-AI transcripts, these platforms prove valuable not just for keeping teachers central to instruction, but for managing the burdensome task of reviewing dialogue logs. Automated diagnostic tools highlight specific conversational pivot points where a student succeeded or struggled, allowing teachers to zero in on genuine teachable moments.
Schools frequently ask how they can technically prevent students from using AI to circumvent deep thought. A far more productive question is how educators can design assignments where student reasoning remains entirely visible. While no system is completely cheat-proof against a determined student, live and adaptive conversations judged on the quality of real-time reasoning leave far less room for outsourcing thought.
Ultimately, artificial intelligence makes it possible to construct assessments that mirror real-world dynamics: a confused peer, a skeptical reviewer, or an unyielding colleague. By forcing students to actively employ core academic skills to navigate those conversations—while providing teachers with a verifiable record of the process—schools can move past the limitations of passive tutors waiting silently at the edge of a screen.