Title: AI-Enhanced Chatbot to Improve Personalized Learning in a College Course
Author Name: Briana Dayton
Selected Case (Published Article): Meyer, Katharine, Lindsay C. Page, Eric Smith, B. Tyler Walsh, C. Lindsey Fifield, and Michael Evans. (). Let’s Chat: Chatbot Nudging for Improved Course Performance. (EdWorkingPaper: 22-564). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/es6b-sm82
1. Introduction:
Text nudges, or encouraging messages via cell phone text, have demonstrated impact in shifting college student behavior (Hall 2022). Artificial-Intelligence (AI) enhanced text communication (“chatbots”) have shown positive results in aiding college students in enrolling and completing tasks like financial aid forms. Georgia State University has used an AI chatbot named “Pounce” for such tasks with demonstrated positive results for several years now (Meyer et al, 2022). Text nudges have shown increased completion of forms like the FAFSA, but there has been less research on the use of text nudges within courses for academic purposes. There is much interest in how to support students within college courses to improve academic achievement, particularly in courses that have traditionally high non-pass rates. Nudges using emails or messages within software (such as within an online textbook or within a course learning management system) have shown positive impact on academic behavior like completing assignments and submitting assignments on time (Hall 2022). However, there is also research that demonstrates that emails or texts perceived as unconnected to a live, known human, like a professor or a counselor, as less impactful on changing student behavior (Hall, 2022). It is not enough to send an email or series of emails, as traditional college-age students have demonstrated less interaction and responsiveness to emails; it is not enough to send a text or a series of texts without a crafted persona and live response ability, as students also quickly ignore those texts just as they do emails. It is important to develop a text with a persona and a live response capability.
Thus, the question was posed, can text nudges sent by a trusted professor or TA within an academic course improve academic outcomes? This randomized control trial sought to expand Pounce at GSU to aid students within a college course with a high non-pass rate to see if there was demonstrated positive impact on student achievement within the course.
2. Overview of the Case:
A single course was selected for this randomized control trial (RCT). “Introduction to American Government” was the course; this is a large, online, required course that many first-year students take at GSU. In this study, half the students were in their first year and more than eighty percent of the other half were in their second year. This course traditionally has a high non-pass rate. This course was unique in that the instructor of the course is a tenured professor (many of these courses are taught by lecturers and graduate assistants, who are in the role for only a semester or two) who has taught this course for several years. The Professor has structured the course with an online textbook that can track reading time and automatically assess reading assignments, and the Professor also already sends weekly emails with assignment reminders; in addition, there was an assigned Teaching Assistant who ran the intervention as well as Supplementary Instruction tutoring sessions. Thus, the structure of the course was set and there were mechanisms in place for observation, communication, and formative feedback to students, and the system was personalized with the Professor and the Teaching Assistant.
3. Solutions Implemented:
The RCT was that half of the students were randomly selected to receive the intervention texts and half did not receive the intervention texts (the half who did not receive texts still received emails from the Professor and the TA, just not texts). The AI-enhanced chatbot text communications via “Pols Pounce,” named for the course section of Political Science, were sent 2-3 times per week to students in the study, for about forty total text messages during the semester. The texts were personalized with the students’ names and tailored to their current performance, i.e. whether they had turned in an assignment or not, and reminded them of upcoming tasks like tutoring sessions and exams as well as sent encouraging messages. Messages were sent each Monday to preview the week’s assignments and were custom tailored to the student’s performance in the previous week, i.e. whether they had completed the assignments in the previous week on time. Students were able to text back and receive a response either immediately from the AI or within the day from the TA if the AI had not been programmed with the answer yet. This study was conducted in the fall of 2021, which is also a time period in which students had experienced a radical shift in education in the past few years due to COVID-19 school shut-downs and moves to online courses.
4. Outcomes:
The students who received the POLS Pounce messages were eight percentage points more likely to receive a final course grade of B or higher than students who did not receive the POLS Pounce messages. First-generation college students were twelve percentage points more likely to earn a final course grade of an A, sixteen percentage points more likely to earn a final course grade of B or higher, and thirteen percentage points more likely to pass the course than first-generation students who did not receive the text messages. Ninety-two percent of students who received the messages reported on an exit survey that the chatbot should be continued in the course and expanded to other courses. However, a limitation to this final finding of support for POLS Pounce is that only fifty-two percent of students who received the messages completed the exit survey.
5. Implications:
AI-enhanced chatbots require an initial investment to purchase and at least one person to write initial messages and craft responses to teach the AI. This initial investment of time and money can be substantial. However, the impact that the AI chatbots can have quickly grows as the AI learns and fewer unique responses are required from a person, which allows for the person monitoring the AI to expand the AI to other functions. The use of something like a chatbot in a course also requires that the course is structured ahead of time so that messages about assignments and performance on them can be sent in a timely way; this course had an online textbook and some assignments that were automatically graded with course software. In a course with less structure and dependent upon humans grading assignments with some subjectivity or that require human feedback, the use of a chatbot might be far less timely and thus less impactful. Additionally, this course had a person dedicated to responding to the AI flagged messages (the messages the AI was not yet programmed to answer) within the day and that person was connected to the course as a TA, so the learners were familiar with the person. The structure, familiarity, and investment in the technology were all already in place in order to set up this RCT.
At the same time, the need for redesigning the educational experience to meet today’s learner is in a critical state, as technology has changed and people have shifted their lifestyle and learning habits. This study, and related studies such as an upcoming expansion of Pounce to other courses (an RCT on economics is forthcoming), are important contributions to the literature on how to personalize messaging and leverage technology in academic settings. This study adds to the literature that AI-enhanced chatbots can improve outcomes in higher education, and that personalizing the message and the sender is critical; that this type of text messaging has an impact beyond email messaging; and that immediate responses are an important component of chatbot success. The implications are that we should continue to study AI-enhanced tools for personalized and rapid communication within academic settings.
References:
Hall, Ann L.. “Text Nudges: Proactive Communication by an Urban Community College.”
(2022). https://digitalrepository.unm.edu/educ_teelp_etds/359