Title: Simulation-Based Learning and the Rise of Teachlive / Mursion Mixed Reality Platforms
Author Name: Erin Anderson
1. Introduction:
Simulation-based learning has been used in the fields of healthcare, medicine, aviation, and recently, the interest has expanded into other fields like education and teacher training. Chernikova et al. (2020) showed that while in-person teacher residencies are used as an opportunity for pre-service teachers to gain real-world experiences, this can engender poor results without proper guidance that can result in ethical and moral issues when an unprepared teacher is put in charge of a group of students (pp. 2). Furthermore, they showed that these in-person experiences do not always provide pre-service teachers the opportunity to engage in critical situations that rarely surface during their student-teaching experience (pp. 3). Also, teachers are reluctant to give student teachers larger roles because they themselves are held accountable for their student outcomes, not the student-teacher (Deiker et al., 2008). Therefore, simulations can be used to allow pre-service teachers the opportunity to engage in practice amid carefully controlled situations. Chernikoya et al. (2020) compiled a meta-analysis of simulations conducted in the fields of medicine, nursing, psychological counseling, management, teacher education, and areas like engineering and economics found that simulations have significant positive effects on learning complex skills, even when compared to “real” controls (pp. 24) and those simulations with high authenticity have the highest impact (pp. 25).
The rise of (X)reality, a.k.a. virtual, mixed, or augmented reality, has paved the way for new types of learning simulations, including within the realm of teacher education. Dieker, Hynes, Hughes, and Smith (2008) at the University of Central Florida, in partnership with NASA and Disney, created a mixed-reality platform to train teachers in a virtual classroom with student avatars which they trademarked Teachlive. According to Deiker et al., “Unlike research in actual classrooms where controlled data collection is difficult to ascertain, this virtual environment enables consistency in preparation, immediate feedback, and ongoing data collection, as well as refinement of the environment, to ensure the maximum impact on teacher performance and student learning (pp. 5). The researchers used the American Academy of Child and Adolescent Psychiatry’s stages of child development, William Long’s classification of children’s personalities, and Rudolf Dreikurs’ 4 goals of misbehavior, to develop the student avatars (pp. 7-9). A human interactor then uses various technologies to control the avatars, acting out their various personalities throughout the simulation (pp. 10).
As time progressed, the Teachlive research team opened their technology to other universities. For example here at Georgia State University, at the mixed reality lab created by Dr.Donehower and Dr.Jiminez, we use their technology to conduct our own research (Hillarygmeister, 2019). Various programs, from special education to school counseling, have used this mixed-reality technology to conduct simulation-based research.
2. Overview of the Case
The apparent effectiveness of the simulations has led to increasing interest in this mixed reality technology. As Dieker, Hynes, Hughes, and Smith (2008) note, the University of Central Florida was perfectly positioned to develop this technology due to a 40-year partnership with organizations like NASA, Lockheed Martin, Siemens, Disney, and Universal (pp. 2). The initial prototype of the Teachlive mixed reality program began as TeachMe, created in partnership with Haberman Education Partnership and Simioysis LLC and it focused on helping pre-service teachers manage student behavior (pp. 4).
Nagendran, Pillat, Kavanaugh, Welch & Hughes (2014) were the developers of the software and they chose a mixed reality platform because artificial intelligence wasn’t developed enough to provide realistic student responses which negatively impacted bidirectional conversational flow and resulted in a reduced sense of authenticity for participants in the simulation (pp. 110). Their AMITIES software system, which stands for Avatar-Mediated Interactive Training and Individualized Experience System, is a combination of custom digital puppetry, a low-cost user interface, autonomous avatar behaviors, a network protocol that supports interaction despite low connectivity, and a storage and retrieval system that allows users to record and playback their participation within the simulation (pp. 113). They utilize a human-in-the-loop that controls the avatar, which was inspired by the Wizard-of-Oz technique that combines traditional methods, simple AI techniques, and a human inhabiter (pp. 112). While the interface has pre-programmed personalities and movements for the student avatars, their personalities can be adjusted after discussion with the researchers and the interactor hired to play the avatars (pp. 117). To control the actions of the avatars, the interactor initially used Razer Hydra technology and joysticks to modulate the behaviors of the avatars (pp. 124). The researchers settled on this student-avatar / human-in-the-loop format after testing participants’ preference on practicing college interviews face-to-face, face-to-face with a 2D virtual character, or face-to-face with a physical mannequin designed to act like a person where all participants preferred the 2D virtual character interaction (pp. 129). This preference for the mixed reality platform has been validated throughout the years and most recently in a study that showed participants not only learned more in the MR simulation than face-to-face, but felt that their learning was more generalizable to their academic field (McKown et al,. 2021). Participants also have the opportunity, as they stand before their screen interacting with the student avatars, to be observed by researchers/facilitators / other participants who can provide feedback on their performance (Nagendran et.al, pp. 127, 2014). However, this technology is primarily a research technology to be employed by the University of Central Florida. As demand for the technology increased, so did a need to allow it to step into the corporate world and take on a life of its own.
3. Solutions Implemented
As a post-doctoral student at UCF, Arjun Nagendran created the mixed reality AMITIES software which powered Teachlive, and in 2015, he partnered with a San Francisco-based serial entrepreneur Mark Atkinson to create the Mursion technology and expand its access (Fink, 2021). The company is now valued over $40 million with over 150 employees (Mursion company, 2021).
When I explored the possibility of licensing Mursion technology for Georgia State University in September, I spoke to a business representative for Mursion. I learned that Mursion has since expanded beyond the educational field to create scenarios for soft-skill development in fields like healthcare, business, and defense. Users can choose from a variety of pre-designed sets and characters to create simulation scenarios. For education, they have over 55 pre-designed scenarios.
Universities that purchase their software can have researchers/ administration make an appointment through a portal with Mursion to schedule a time for their participants to interact with the avatars. Researchers work with the interactors beforehand to ensure fidelity of the scenario scripts. The business representative recommended white labeling the Mursion software and then developing the lab to become a vendor for other interested agencies in the area. For example, an institution that white-labeled Mursion could then charge other institutions a fee to come into their labs and develop scenarios. For an additional cost, Mursion would even take over the development of the lab and the hiring and training of an in-house simulation specialist that will not only control the avatars but be responsible for the management of appointments among researchers and institutions. As an example of this, the representative directed me to check out the California State University at Northridge’s simulation lab, which uses Mursion software, called SIMPACT. It’s a great example of a university that white-labeled Mursion and makes money for the university by letting other organizations come into the lab to host simulation scenarios.
Over the next few years, according to the analyst, Mursion hopes to roll out a data analytics system that uses AI, facial recognition, and eye-tracking technology to print out a report for participants to better understand their nonverbal behavior while in the simulation.
4. Outcomes
Both Mursion and Techlive have produced research in a variety of fields. It has been used to develop pre-service teachers’ efficacy when delivering feedback (Bardach et al., 2021). It has been used to do diversity training with geoscientists (Chen at al., 2020). It has been used to measure pre-service teachers’ implicit bias (Collum et al., 2020). It has been used to train counselors, recruit students to STEM, prepare Algebra teachers, conduct discrete-trail-trainings, teach young adults cognitive job skills, and even investigate teachers’ perception of Latinos who are labeled emotionally disturbed (Deiker et al., 2014).
As of 2021, Mursion even uses GPT-3, computer vision, motion capture, AI-assisted body language, and voice morphing to portray its avatars (Fink, 2021).
As this program scales and competitors come onto the market, researchers Bondie, Mancenido, and Dede made 5 recommendations for principles that can guide virtual humans in the execution of these mixed-reality scenarios for teacher training (2021). First, they recommended detailed documentation of the learning design process to ease replicability in future studies. Secondly, they recommended measuring not only teacher response and learning but changes in the quality of avatar responses. Thirdly, they recommended detailed documentation of training and actions of the interactor, with attention paid to the interactor’s race, language, and culture. Furthermore, they recommended extending the time of the studies to ensure transferability of the skills into real-world application. Finally, the researchers stressed the need to contextualize the learning activity amid broader socio-cultural implications. For example, presuming the neutrality of the design process risks neglecting potential bias that could be embedded in the development and execution of the simulation scenarios.
5. Implications
There is great promise in the use of XR technology, like Teachlive and Mursion, to use simulations to in the educational process. Interests in transformative learning, authentic learning, and problem-based learning are all applicable to XR simulations. As this technology scales and becomes cheaper to adopt, one could imagine a world where all schools have such simulation technology, to help students develop soft skills. As marketed on the Mursion blog, skills like leadership, team work, communication, problem-solving, work ethic, flexibility, and interpersonal skills are what employers will be valuing as many processes become automated. Mursion reality can provide a safe space to practice such skills and be prepared for the job market (What makes you human? why soft skills are actually human skills, 2021). Imagine having a tough conversation coming up that you’d like to rehearse beforehand. Or imagine practicing an elevator pitch beforehand and receiving immediate feedback. In the future, mixed reality simulation technology could be what we use to do this.
However as with the development of all XR technology, the potential for bias to creep into the design must vigilantly be addressed. Researchers Bondie, Mancenido, and Dede (2021) already warned about this possibility in their piece Interaction principles for digital puppeteering to promote teacher learning. Furthermore, as AI develops and many of the actions of the interactors become automated, the problems with biased AI could easily be replicated within the development of the technology. Ruha Benjamin warns about this in her book Race After Technology where she systematically showed how machine learning algorithms have actually hurt marginalized people and perpetuated the development of the New Jim Code. She defines the new Jim Code as “the employment of new technologies that reflect and reproduce existing inequities but that are promoted and perceived as more objective or progressive than the discriminatory systems of a previous era” (Benjamin, pp.8, 2019).
As simulation technology trickles from academia and corporations down into schools and eventually into our homes, it has the possibility to drastically shift how we go about learning endeavors. With Zuckerberg’s announcement of his desire to dump billions of dollars into the creation of the Metaverse, XR technology is currently having its moment. Sam Altman, the CEO of OpenAI and Y Combinator, tweeted on November 29th, “If you are working in fields like AI or web3 or VR right now, you are living through and creating important history. Enjoy it and keep notes. May victory be yours.”
For more information, check out:
2. xR, AR, VR, MR: What’s the Difference in Reality?
3. Interview with Mursion Simulation Interactor
4. Mursion Virtual Classroom ABC Los Angeles
REFERENCES
Bardach, L., Klassen, R. M., Durksen, T. L., Rushby, J, K., Bostwick, K. C. P., & Sheridan, L. (2021). The power of feedback and reflection: Testing an online scenario-based learning intervention for student teachers, Computers & Education, 169, 104194.
Benjamin, R. (2019). Race after technology: Abolitionist tools for the new Jim code. Polity.
Bondie, R., Mancenido, Z., & Dede, C. (2021). Interaction principles for digital puppeteering to promote teacher learning. Journal of Research on Technology in Education, 53(1), 107-123.
Bursali, H., & Yilmaz, R. M. (2019). Effect of augmented reality applications on secondary school students’ reading comprehension and learning permanency. Computers in Human Behavior, 95, 126-135.
Chen, J. A., Tutwiler, M. S., & Jackson, J. F. (2020). Mixed-reality simulations to build capacity for advocating for diversity, equity, and inclusion in the geosciences. Journal of Diversity in Higher Education.
Chernikova, O., Heitzmann, N., Stadler, M., Holzberger, D., Seidel, T., & Fischer, F. (2020). Simulation-based learning in higher education: a meta-analysis. Review of Educational Research, 90(4), 499-541.
Collum, D., Christensen, R., Delicath, T., & Knezek, G. (2020, April). Measuring changes in educator bias in a simulated learning environment. In Society for Information Technology & Teacher Education International Conference (pp. 507-513). Association for the Advancement of Computing in Education (AACE).
Dieker, L., Hynes, M., Hughes,C., & Smith, E. (2008). Implications of mixed reality and simulation technologies on special education and teacher preparation. Focus on Exceptional Children, 40(5),1-20.
Dieker, L. A., Rodriguez, J. A., Lignugaris/Kraft, B., Hynes, M. C., & Hughes, C. E. (2014). The potential of simulated environments in teacher education: Current and future possibilities. Teacher Education and Special Education, 37(1), 21-33.
Fink, C. (2021, June 7). Mursion’s Human Powered Tech Stack. Forbes. Retrieved December 1, 2021, from https://www.forbes.com/sites/charliefink/2021/06/07/mursions-human-powered-tech-stack/?sh=63d3297b1811.
Hillarygmeister. (2019, August 23). Interactive teaching and Learning lab (ITLL). College of Education & Human Development. Retrieved November 25, 2021, from https://education.gsu.edu/2019/08/23/interactive-teaching-and-learning-lab-itll/.
McKown, G., Hirsch, S. E., Carlson, A., Allen, A. A., & Walters, S. (2021). Preservice special education teachers’ perceptions of mixed-reality simulation experiences. Journal of Digital Learning in Teacher Education, 1-16.
Mursion Company Profile: Valuation & Investors. PitchBook. (n.d.). Retrieved December 1, 2021, from https://pitchbook.com/profiles/company/111236-77#overview.
Nagendran, A., Pillat, R., Kavanaugh, A., Welch, G., & Hughes, C. (2014). A unified framework for individualized avatar-based interactions. Presence: Teleoperators and Virtual Environments, 23(2), 109-132
What makes you human? why soft skills are actually human skills. Virtual Reality Training Simulation Software by Mursion. (2021, April 29). Retrieved December 1, 2021, from https://www.mursion.com/blog/why-soft-skills-are-actually-human-skills/.