Title: The effect and learner experiences of using AI for creative writing
Author Name: Chakrabarty, T., Padmakumar, V., Brahman, F., & Muresan, S.
Selected Case (Published Article): Creativity support in the age of large language models: An empirical study involving professional writers. Proceedings of Creativity and Cognition
1. Introduction
This study examines the role of AI, particularly ChatGPT, in scaffolding the creative writing process by analyzing the experiences of professional writers. Grounded in the Cognitive Process Theory of Writing (Flower & Hayes, 1981), the study highlights the dynamic interactions among a writer’s long-term memory, the writing process, and task-related factors, emphasizing non-linear and adaptive thinking.
Existing research presents mixed findings on the efficacy of AI as a writing scaffold. Shidiq M. (2023) identified several negative impacts of ChatGPT in educational contexts, such as the lack of instructor interaction, difficulty generating original ideas, insensitivity to nuances and diverse learning styles, reduced social interaction, and diminished critical thinking skills. Similarly, Niloy et al. (2024) found that while ChatGPT improved “presentability” and “elaboration,” it negatively impacted “similarity” and total scores, raising concerns about its influence on creativity. Conversely, Plate and Hutson (2022) observed that AI tools effectively reduced syntax and grammar errors while aiding concept comprehension. However, students expressed skepticism about AI’s ability to enhance creativity, reporting negative perceptions. Lastly, Ippolito et al. (2022) explored professional writers’ experiences with Google’s Wordcraft program and found AI helpful for brainstorming, world-building, and research assistance. However, limitations included the inability to personalize tone, handle nuanced storytelling, and deeply understand content.
These findings highlight a spectrum of perspectives on AI’s role in creative writing. While AI shows promise in refining technical aspects and providing brainstorming support, challenges persist in fostering creativity and adapting to individual styles. Further research is needed to identify key features, evaluate educational impacts, and analyze learners’ experiences in AI-supported environments.
2. Overview of the Case
This study evaluated the effectiveness of large language models (LLMs), particularly GPT-3.5, in supporting professional writers during the creative writing process. Seventeen MFA (Master of Fine Arts) students co-authored 30 short stories across various genres using a custom interface designed to integrate AI assistance during the Planning, Translation, and Reviewing phases. Participants provided qualitative feedback through a post-writing survey to assess their experiences.
3. Solutions Implemented
The AI-supported interface was developed based on the Cognitive Process Theory of Writing and emphasized the non-linear nature of writing. Three core features supported the writing process:
- First Draft Generation: Writers provided a plot and genre, and GPT-3.5 generated an initial story draft.
- Interactive Editing: Writers interacted with a chatbot, using natural language instructions for revisions, such as “Rewrite this passage with richer imagery” or “Insert a dialogue between characters.”
- Templated Prompts: Pre-designed prompts guided tasks like elaborating on details, rewriting with enhanced imagery, and offering feedback on drafts.
This design identifies tasks within the writing process that AI can scaffold and implements scaffolding through interactive communication between humans and AI via chat.
4. Outcomes
The study revealed mixed results regarding GPT-3.5’s utility in creative writing. Writers found AI particularly effective in the Translation and Reviewing phases, aiding in rewriting and refining passages, as well as ensuring logical consistency. However, its utility in the Planning phase was limited, as AI-generated suggestions were often generic or uninspired. Challenges included reliance on clichés, overly moralistic and predictable story endings, and struggles with darker or complex themes due to safety alignment protocols.
Outcomes demonstrated that writers retained less than 35% of AI-generated drafts in their final versions, suggesting that AI served more as a brainstorming tool than a content creator. Writers emphasized the need for improved style adaptation, enhanced creative risk-taking, and support for proper citations. These findings underscore both the potential and limitations of AI in creative writing.
5. Implications
This study provides valuable insights for future research, analyzing how AI supports professional writers, its effectiveness, and writers’ perceptions. Participants recognized AI’s role in logically refining ideas, while highlighting its limitations in generating original concepts and addressing tone or nuance. These observations relate to broader discussions on agency, as writers’ perceptions of AI’s contribution influenced whether a work was considered a product of human creativity.
Future research should adopt a constructivist paradigm, viewing writing as a process of constructing meaning at individual and societal levels. While this study used a cognitive framework focused on linguistic and task-related features, integrating constructivist theories could provide deeper insights into scaffolding’s role in addressing challenges like generating new ideas and handling nuanced styles.
Moreover, advancements in AI technology since GPT-3.5 may enable more sophisticated capabilities, offering further opportunities to explore its educational potential. Future studies should investigate how evolving AI tools can enhance creative writing instruction and foster meaningful collaboration between human creativity and machine learning.
References
Chakrabarty, T., Padmakumar, V., Brahman, F., & Muresan, S. (2024). Creativity support in the age of large language models: An empirical study involving professional writers. Proceedings of Creativity and Cognition (C&C ’24), June 23–26, 2024, Chicago, IL, USA. ACM. https://doi.org/10.1145/3635636.3656201
Flower, L., Hayes, J. R., (1981). A cognitive process theory of writing. College composition and communication 32, 4, 365–387.
Ippolito, D., Yuan, A., Coenen, A., & Burnam, S. (2022). Creative writing with an AI-powered writing assistant: Perspectives from professional writers. Google Research. https://doi.org/10.48550/arXiv.2211.05030
Niloy, A. C., Akter, S., Sultana, N., Sultana, J., & Rahman, S. I. U. (2024). Is ChatGPT a menace for creative writing ability? An experiment. Journal of Computer Assisted Learning, 40(2), 919–930. https://doi.org/10.1111/jcal.12929
Plate, D., & Hutson, J. (2022). Augmented creativity: Leveraging natural language processing for creative writing. Art and Design Review, 10(3), 376–388. https://doi.org/10.4236/adr.2022.103029
Shidiq, M. (2023). The use of artificial intelligence-based ChatGPT and its challenges for the world of education: From the viewpoint of the development of creative writing skills. International Conference on Education, Society and Humanity, 1(1). https://ejournal.unuja.ac.id/index.php/icesh