Case Study Title: The Impact of Generative AI on Student Learning
Selected Case (Published Article): Generative AI Can Harm Learning
Authors Name: Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge,3 Ozge Kabakcı, Rei Mariman
1. Introduction:
Generative AI tools, like GPT-4, are becoming a common feature in education, helping students and teachers perform tasks more efficiently. These tools show great promise in improving productivity, but there’s a key question we still need to address: how do these tools affect the way students learn over time? Are students truly gaining knowledge, or are they simply completing tasks faster without mastering the skills they need?
This case study specifically looks at how GPT-4-based AI tutors impact student learning in mathematics. The question to be answered: Can AI tools improve performance without sacrificing long-term learning?
a. Objective: The case explored a large-scale trial at a high school in Turkey to test how different types of AI aid influence student learning. It focused on whether students who used GPT-4 for tutoring performed better on practice problems and exams compared to those who didn’t use AI resources.
2. Overview of the Case:
Generative AI, particularly OpenAI’s GPT-4, is transforming education by enhancing productivity. However, a critical question remains: how does it affect skill development? This idea is especially important in domains where AI may make mistakes, requiring human intervention to check its outputs.
This case detailed a large-scale field experiment with nearly 1,000 high school students who tested two versions of GPT-based tutors: GPT Base, which mimics a standard ChatGPT interface, and GPT Tutor, designed with prompts to safeguard learning. These tutors were used in about 15% of the curriculum across three grade levels.
The results show that GPT-4 significantly improved performance—48% with GPT Base and 127% with GPT Tutor. However, when students were later asked to work without AI, those who used GPT Base performed 17% worse than those without access to AI. This event suggests that, while generative AI boosts short-term performance, it may hinder long-term learning if students become reliant on it.
The negative effects were disappointing with GPT Tutor, which was designed to encourage deeper engagement with the material. This fact highlights the importance of thoughtful AI integration in education to support skill development while enhancing learning outcomes.
3. Solutions Implemented: To address the potential drawbacks of overreliance on generative AI while still fostering effective learning, two different versions of GPT-4 were tested:
a. GPT Base: This version provided students with access to a standard ChatGPT interface, allowing them to use the AI for solving practice problems. While it was effective in helping students complete tasks more quickly, it didn’t focus on promoting deeper learning or skill retention.
b. GPT Tutor: In contrast, this version was specifically designed with built-in prompts to encourage critical thinking and ensure that students were engaging with the material. It was tailored to foster active learning and help students build understanding, rather than just completing tasks. By incorporating these safeguards, GPT Tutor aimed to prevent students from becoming overly dependent on the AI.
4. Improved Performance on Practice Problems:
a. Students using the GPT Tutor (with built-in safeguards) saw a 127% improvement in their ability to solve practice problems compared to those who didn’t use AI at all. This event shows that a well-designed AI tool can really help boost problem-solving skills.
b. Students with GPT Base (the standard ChatGPT interface) saw a 48% improvement, indicating that simply having access to AI can make a noticeable difference in their learning .
5. Impact on Exam Results:
When it came to exams, however, the results were more mixed. Students who used GPT Base actually performed 17% worse than those who didn’t have access to AI. This fact suggests that while GPT-4 helped them practice, it may have hurt their ability to do well on their own later.
On the other hand, the students using GPT Tutor—which was designed to be more supportive of learning—didn’t show a decline in exam scores, though they didn’t see a boost either. Outcomes:
The use of GPT-4-based tutors led to some interesting results in terms of student performance, with both positive and cautionary insights:
a. Implications:The findings of this study highlight several important considerations for the use of generative AI in educational settings, especially in relation to its potential effects on long-term learning and skill acquisition.
6. Balancing AI Support with Skill Development:
One of the most significant takeaways is the need to balance AI assistance with students’ ability to independently acquire and apply skills. Although accession to GPT-4 improved immediate task performance, the removal of this support led to worse outcomes. This response suggests that overreliance on AI can undermine deeper learning. Hence, it is crucial to integrate AI in a way that enhances student productivity without hindering their capacity for independent problem-solving and critical thinking.
7. AI Design for Learning Enhancement:
The contrast between the GPT Base and GPT Tutor models highlights the importance of designing AI tools that do more than just provide answers. While the GPT Base model improved performance, it did not effectively support skill retention or learning. In contrast, the GPT Tutor model, which incorporated prompts to encourage deeper engagement, led to better overall outcomes. This issue suggests that AI tools should be carefully designed to promote active learning—encouraging students to understand and internalize the material rather than simply offering solutions.
8. The Role of Educators in AI-Enhanced Learning:
The results also reinforce the pivotal role of educators in guiding AI use. Teachers should provide structured support, ensuring that AI tools are used strategically to reinforce learning rather than replace traditional learning methods. In this context, educators are essential in creating a learning environment that fosters both immediate performance and long-term skill development.
9. Training for Efficient AI Use:
As AI becomes more integrated into educational environments, it is vital that students are educated on how to use these tools effectively. This idea involves teaching students when and how to rely on AI, while also encouraging them to work through problems on their own. A well-rounded approach to AI training can help students avoid the temptation to use AI as a shortcut and instead leverage it as a supplemental tool to deepen their understanding.
a.Considerations for Long-Term Skill Development:
i. The study raises broader questions about the long-term impact of AI on skill development. While AI can enhance performance in the short term, it is essential to ensure that students continue to develop the core skills necessary for success in their academic and professional lives. If students become too dependent on AI, they may miss out on the foundational skills that are critical for their future learning and career growth.
10. Need for Farther Explorations:
While this study provides valuable insights, it also suggests that further research is necessary. Future studies should explore the effects of AI on learning across different subjects, age groups, and educational contexts. Investigating how AI can be best integrated into different learning environments and how it can be tailored to different learning needs will be crucial for maximizing its effectiveness.
- Conclusion:
Generative AI offers significant potential to enhance learning outcomes, its use must be carefully managed to avoid detrimental effects on long-term skill acquisition. By ensuring that AI tools are used to complement, rather than replace, traditional learning methods, educators can create a more balanced approach that promotes both immediate performance improvements and the development of critical skills. Further research and thoughtful design of AI tools will be essential for realizing the full potential of AI in education.
References:
Bastani, H. (n.d.). Generative AI Can Harm Learning. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486
OpenAI. (2025, April 22). Generative AI and its impact on human learning: A field experiment with GPT-4 in high school math classes. OpenAI. https://www.openai.com/chatgpt
Schneppat, J.O. (2019) Daphne Koller & AI, Schneppat AI. Available at: https://schneppat.com/daphne-koller.html (Accessed: 27 April 2025).