AI Tutors Can Work—With the Right Guardrails

On its surface, AI seems like a poor teacher.

In a 2024 study from the University of Pennsylvania, 1,000 high school math students were given a math lesson, then asked to solve a set of post-lesson practice problems with the aid of either their class notes or ChatGPT. The students who had access to ChatGPT did 48 percent better—but when it came time for a closed-book (and closed-laptop) follow-up test, their performance plummeted to 17 percent below their low-tech peers.

The researchers found that among students who had used ChatGPT to aid them with the practice problems, a third of their interactions with the bot had consisted of their asking some variation of “What is the answer?” With artificial intelligence (AI) serving up solutions, it’s unsurprising that these students saw a significant decline in their ability to solve math problems themselves.

But AI, it turns out, can be a useful tutor—if it’s designed and deployed with the proper guardrails.

That 2024 study included a third group: Students who had access to a “tutor” version of ChatGPT that researchers had customized “with prompts designed to safeguard learning.” The researchers provided the AI with “teacher input on common student mistakes and how to provide feedback,” and prompted it to “provide hints to the student without directly giving them the answer.”

Students who had access to this AI tutor scored a whopping 127 percent higher on the initial problem set than the group of students who did not have access to ChatGPT—and scored roughly the same on the closed-laptop exam as this low-tech group.

This study is part of a growing body of research suggesting that allowing students to use AI base models unsupervised and without restrictions in place can hinder student performance, but well-designed AI tutors may be able to bolster learning—similar to the effect that human tutors can have on children.

Here’s a brief overview of the findings—and some tips on how teachers can customize their own AI chatbots for students to interact with.

AI Tutors, Real Impact

In some sense, AI tutors prove more useful for students for the exact same reasons that high-quality human tutors have always been effective. “When you have that one-on-one instruction, or small group instruction, the instruction caters to your needs,” Sal Khan—founder of Khan Academy—told us in November. “If a student is finding something easy, then a tutor can move ahead or go deeper. If they’re struggling, a tutor can slow down.”

In another study from last year, an AI-powered tutor helped Harvard undergraduates learn physics more effectively—and in less time. To optimize the bot for learning, the researchers prompted it with a set of constraints—including “Only give away ONE STEP AT A TIME, DO NOT give away the full solution in a single message,” and “You may CONFIRM if their ANSWER is right, but DO NOT tell [students] the answer.”

One hundred and eighty-six physics undergrads were split into two groups: Half received a standard physics lecture, and the other half stayed home to work through the same material under the guidance of the AI tutor. When the researchers compared students’ scores on a pre- and post-lesson assessment of the material, “the learning gains for students… in the AI-tutored group were over double those for students in the active lecture group.” The AI tutor group also worked through the material 10 minutes faster than their peers—and reported higher levels of engagement and motivation. 

These findings align with a 2023 review of two dozen studies, which found that having students work through material with an AI chatbot is linked to increases in learning performance, motivation, and interest, as well as a decrease in anxiety.

None of this is to say that AI is a better teacher than a human—far from it. “We believe that the increased learning from AI tutoring is largely due to its ability to offer personalized feedback on demand—just as one-on-one tutoring from a (human) expert is superior to classroom instruction,” the Harvard researchers write.

For that reason, if you do plan to use AI in your classroom, it may be helpful to find ways to give your students a structured experience with an AI tutor, rather than simply allowing them to use chatbots without restrictions.

How to Create an AI Tutor

While the field is in its infancy, there are already some chatbots on the market specifically designed to align with educational best practices.

One of the most prominent examples is Khanmigo, an AI tutor developed by Khan Academy. Khanmigo is split between two different tutors optimized for different subjects—one for math and science, and the other for the humanities. As Khan Academy shows in its demo, both have been designed not to give students the answers. Instead, “it’s going to try to ask you questions, understand where you are, and then help you fill in any gaps in your understanding.” If a student prompts it with “I want to understand photosynthesis better,” it won’t respond with a bunch of information, but rather with friendly, probing questions like “What do you think plants need to perform photosynthesis?”

Students can access Khanmigo for $4 per month—and Khan Academy also offers a free but limited version of the bot, Khanmigo Lite, on the ChatGPT platform.

But it’s also possible for teachers to create their own, customized AI tutors. Premium subscribers to ChatGPT can create a “custom GPT” that will adhere to a particular prompt across all its interactions; once it’s made, other ChatGPT users (even those on a free account) can chat with the custom GPT by being sent the link. Teachers can also create a customized tutor for free on MagicSchool AI—a platform that has become quite popular among our community. The whole process takes about five minutes.

To start, after signing in, teachers can go to MagicSchool’s “Rooms” page and click on “Create New Room.” While setting up the room, under “Tools,” search for “Custom Chatbot” and add it to the room. Then, during the “Configure” step, click again on the custom chatbot in order to provide it with unique instructions. These instructions will persist throughout all future conversations that the chatbot has. You can find our sample prompt, which we refined over several practice sessions, at the bottom of this article.

When providing these custom instructions, teachers can mention the age level and subject area of the students who will be using the chatbot, and they can prompt the bot to adhere to certain best practices, like “asking probing questions that will help students reach a conclusion” and “NEVER providing the answer directly.”

During set-up, teachers have the option to hide this prompt to prevent students from seeing or altering it when they use the bot for themselves. If there are documents that would be useful as relevant background information, teachers can upload these in the “Knowledge” section.

After setting up the bot, teachers can launch their new room and send students a link to join it, which they can do for free on their own devices. Once students have joined the room, they’ll see the custom chatbot, and they can immediately begin to have conversations with it.

It’s probably best for students to have their first interactions with the bot in class, rather than at home, so the teacher can troubleshoot any issues they—or the AI—may be having. Having tinkered with the AI themselves, teachers can give students advice about the kinds of messages that elicit the most helpful response from the bot.

Teachers can also give students advice about how to sniff out potential hallucinations—and when to double-check AI outputs with reliable external sources. If the AI is repeatedly deviating from its intended purpose, teachers can try to fix it by going back into the room configuration and updating the custom instructions.

An AI set up in this way will generally do an impressive (but not perfect) job of sticking to its original prompt. Students who sneakily try to get the AI to violate its initial instructions—say, prodding it to start giving them the answers outright—will typically be politely shut down. After students have had the chance to interact with the chatbot in school, teachers may recommend that they try it out at home as a study aid.

With the help of the AI tutor, students can build on what they learned in class—asking questions, spitballing ideas, and answering practice problems provided by the AI. It’s quite possible that only the most dedicated students will actually follow the recommendation—but the research showing that chatbots can decrease anxiety and boost motivation suggests that the appeal might be wider than you’d expect.

Sample prompt:

You are a tutor for the fifth-grade science students in Mrs. Smith’s class. Like a good tutor, you help students come to the right conclusions ON THEIR OWN by asking them probing questions; you NEVER provide them the answer directly. You encourage students to have a growth mindset, and not feel deterred by their mistakes. Keep responses brief—no more than 100 words.

Today’s topic is photosynthesis. Through probing questions, help Mrs. Smith’s students come to a complete and accurate understanding of photosynthesis that answers the following questions:

—Why are leaves green?

—What do plants convert light energy into?

—What role does photosynthesis play in sustaining all life on Earth, not just plant life?

Begin each chat with the question “What do you already know about photosynthesis?” and let the conversation flow from there.