Project Goal: This project seeks to investigate and analyze how students in a first-year college writing course interact with the ChatGPT website/app in the context of educational assignment. From a UX perspective, the researcher wants to understand how the generative AI tool is used, whether students show variations in how they use the tool, and if there might be changes that could improve the experience for student users.
Schedule:
(Completed 3/4/2024) Create and post project outline/schedule
(Completed 3/4/2024) Create questions for focus group discussion
By 3/12/2024: Update assignment instructions
3/12/2024: Provide instructions for student writing assignment that will incorporate ChatGPT usage
3/12/2024: Students create OpenAI accounts
3/12 and 3/14: Students complete the pre-writing/invention assignment
3/14/2024: Students submit links to their ChatGPT conversations
3/19/2024: Conduct focus group discussion
3/20-4/15/2024: Synthesize data and findings; write report (Organizational Structure of Report: Explore, Define, Ideate)
By 4/22/2024: Post final report
Target Research Sample: ~12 ENGL 1101 students at GGC.
GGC student demographic profile:
Application of student demographic information: The researcher believes the institutional data for the college is representative of the student sample he is working with. The researcher has deemed that the benefits of having more accurate demographic data for the research sample is not worth the costs (time and effort) of collecting the data. An attempt to collect this data would be an additional burden and privacy encroachment for the research participants. The risks of not collecting the data are minimal but could result in some problems with the interpretation of certain data that is collected and the accurate representation of the student population of the institution. Generally, the researcher should take into account the diversity of the student population (specifically racial diversity and students with disabilities that impact learning) when designing the case study. Information on student socioeconomic status would be helpful as well because of correlations with access to technology. To that end, 2021 data shows that 62% of GGC students received Pell grants (compared to 47% at comparison group institutions). In sum, GGC serves a student body that is far more diverse and financially in need than other institutions in a comparison group of similar institutions.
https://nces.ed.gov/ipeds/use-the-data
Methods: Students will use ChatGPT for a prewriting activity that will help them explore their own tentative thesis statement or topic.
Prewriting Activity Prompts:
Please help me prepare for writing my persuasive essay about a significant problem for Generation Z. Ask me questions–one at a time–that will help me come to a thesis, understand the strengths and weaknesses of my position and any opposing view(s). The problem I want to focus on is…
–or—
Please help me prepare for writing my persuasive essay about a controversial issue in my field of study. Ask me questions–one at a time–that will help me narrow down my topic to a specific debatable question, come to a thesis, understand the strengths and weaknesses of my position, and understand any opposing view(s).
–or—
Please help me prepare for writing my persuasive essay about a social justice issue. Ask me questions–one at a time–that will help me come to a thesis, understand the strengths and weaknesses of my position and any opposing view(s). The problem I want to focus on is…
–or–
Let’s engage in a dialectic exercise. I will present my tentative thesis, and then you will take the role of Socrates and ask me yes or no questions (one at a time) to expose any potential flaws or contradictions in my position.
Data Collected: Students will be asked to submit their ChatGPT conversations in the form of sharable links provided by ChatGPT. The researcher can use this data for qualitative and quantitative analysis (number of replies and length of replies). Qualitative data will also be collected from a focus group discussion.
Focus Group Discussion Questions:
(Researcher will start with background info on research purpose, discussion of consent, and info about groupthink)
How would you describe your prior use of generative AI (like ChatGPT, claude.ai, etc.)?
Never used it/Used once/Used a few times/Use regularly
For what purposes?
Did you use the mobile app or desktop website? Mobile / Desktop
What were your initial impressions of the appearance of the tool? Design? Functionality? Features?
Notes:
How easy was it to…
find and access ChatGPT and create your account?
Very Easy / Somewhat Easy / Somewhat Difficult / Very Difficult
Comments:
use ChatGPT to….
get responses (any responses—helpful or unhelpful)
Very Easy / Somewhat Easy / Somewhat Difficult / Very Difficult
Comments:
share the link to your ChatGPT conversation?
Very Easy / Somewhat Easy / Somewhat Difficult / Very Difficult
Comments:
What about…
Speed of ChatGPT responses?
Scroll speed of the screen as ChatGPT responds?
Appearance of the interface? Colors, font size? Layout?
Comments:
Additional features
Did you notice the thumbs down button? Did you use it?
Did not notice /Noticed but didn’t use / Noticed and used
Did you notice the message that says, “ChatGPT can make mistakes. Consider checking
important information”?
Noticed / Did not notice
Comments:
Other features you wished the interface had?
Comments:
Features that you wished the interface would change or remove?
Comments:
Accessibility questions
How could the tool be better designed to improve the experience for people with disabilities?
Comments: