Learning Analytics in Higher Education Settings
Chad Marchong
Georgia State University

Abstract
Learning analytics can be leveraged to inform instructional design decisions in higher education courses. The Learning Management System (LMS) and other learning technologies produce a large amount of data (Big Data) from student learning activities. Smith, Lange, and Huston (2012) stated that examples of data from an LMS could be login frequency, course engagement with students and instructors, and assessment/assignment grades. Possible use of Learning Analytics can be the introduction of an intervention to a course. LMS Data from student learning can be analyzed to determine if the intervention was successful in meeting the expected outcome. Another possible use of Learning Analytics can be to inform instructional designers by introducing different interventions to a course and when to introduce the intervention to a course. Data can also be used by instructional designers to advise instructors with pedagogical strategies to address areas of teaching that can be improved. A large amount of data can be analyzed to predict the performance of students throughout the semester and help guide the design of the course to assist with the success of the students.

Keywords: Learning Analytics, Higher Education, Instructional Design

Learning Analytics in Higher Education Settings

Introduction
A definition for learning analytics is “the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Long & Siemens, 2011, p. 32). There is a large amount of data that is generated by learners in learning systems, such as the learning management system. The data can be collected and analyzed to identify meaningful insights about learner patterns. Learning analytics is being adopted at the course and departmental levels to measure course interactions by the learner in the LMS. The data gathered and analyzed are used to support improving learning outcomes for courses and departments. As instructional designers work with instructors and subject matter experts, they will need evidence to make decisions about course design and delivery. Having the ability to access the data about learning can inform more targeted course design and pedagogical strategies. Data can identify when learners are having difficulties with subjects and are expected not to succeed or drop courses. Recommendations about adjustments in course design and deployment of interventions can support students to manage challenges and be successful. Predictive analytics in learning analytics can be used in learning to inform instructors of the progression of learners before issues arise. With this prediction, instructors can intervene with the learners to provide support and resources to persist with learning.

History
The term Learning Analytics first referenced by a learning management system company in 2000 to describe the data that is produced by the LMS. In the late 2000s, big data and data mining allowed for easier access and analysis of a large amount of data. Learning management systems can collect data that is being generated by learner interactions. In the early 2010s, learning analytics conferences, such as Learning Analytics and Knowledge (LAK), and research groups, such as Society for Learning Analytics Research (SoLAR), were created. These initiatives produced research, publications, and best practices for learning analytics.

Importance
There is a vast amount of data that continues to grow with the increased use of technology for teaching and learning. Learning analytics can leverage the data to understand learning higher and inform effective pedagogical strategies. Learning analytics can be used to predict when students may require interventions and support during the semester. Instructional designers can use learning analytics to inform the design of programs and courses to ensure academic success for learners. The U.S. Department of Education stated the following about the use of data in education: “key decisions about learning are informed by data and that data are aggregated and made accessible at all levels of the education system for continuous improvement” (U.S. Department of Education 2010a, p. 35). This statement indicates the importance of using data to inform decisions on improving education in this country.

Theorists and Researchers
George Siemens has been one of the most notable and prominent contributors to the learning analytics field. He has researched and written influential articles about the field. Siemens has provided leadership and guidance about the field while being the founding president of the Society for Learning Analytics Research (SoLAR). Additionally, Siemens has made significant contributions to other learning design and technology concepts, such as the Connectivism Theory and Massive Open Online Courses (MOOCs). While Jean Francois Puget has not directly researched learning analytics, his Analytics Landscape has been applied to learning analytics. Puget’s Analytics Landscape includes four maturity levels for analytics (descriptive analytics, diagnostic analytics, predictive analytics, and prescriptive analytics).

Article: PREDICTIVE MODELING TO FORECAST STUDENT OUTCOMES AND DRIVE EFFECTIVE INTERVENTIONS IN ONLINE COMMUNITY COLLEGE COURSES
This article describes the implementation of a predictive learning analytics model to determine student outcomes. This model was developed to identify students that are at risk of not succeeding in online courses at a community college. The model was built with data from the college’s learning management system. Some of the data that was used in the model included learner logins to the system, logins to the course, accessing of course content, and completing assignments. This data was built into the model to provide the probability of the student being successful in the course. The model included information for faculty members to communicate to students that were predicted to be at risk.

Article: Learning Analytics: The Emergence of a Discipline
Learning analytics is an emerging and growing disciple. In this article, the authors present an overview of learning analytics beginning with the definition, historical roots, and historical contributions. The author describes the different commercial and research tools that can be used for learning analytics. There are references to the learning analytics tools that have been developed and used at institutions. Baker and Yacef (2009) identified five primary areas of analysis for learning analytics and educational data mining that includes predication, clustering, relationship mining, the distillation of data for human judgment, and discovery with models. Bienkowski, Feng, and Means (2012) identified five areas of LA/EDM application that includes modeling user knowledge, behavior, and experience, creating profiles of users, modeling knowledge domains, trend analysis, and personalization and adaptation. The author presents the learning analytics model (LAM) with seven components that include: data collection, data storage, data cleaning, integration, data analysis, representation and data visualization, and action with data. The author describes how learning analytics can be used by instructional designers to create content, interaction, and support resources for learners before they begin the course. Learning design processes can be restructured to utilize data for designing of learning content in the future.

Figure 1. Analytics Model Developed by George Siemens. Reprint from Learning Analytics: The Emergence of a Discipline (p. 1392), by American Behavioral Scientist, 57(10), 2013.
Copyright 2013 by American Behavioral Scientist.

Article: Academic Analytics and Data Mining in Higher Education
Academic analytics and educational data mining can be used to guide course design and the development of assessments. Data can be used to inform the instructor/learner communication. The use of statistical analysis and predictive modeling can lead to changes in academic behavior by learners, instructors, and administrators of educational institutions. Higher education institutes have developed and implemented learning analytics systems to support learning. Purdue University has developed a predictive academic warning system called Signals to identify when learners are at risk and to inform when an intervention should be deployed. The University of Florida used learner activity data to research the predictability of the learners’ sense of community. The University of Auckland has researched online discussions with networking maps that were developed by modeling tools. There has been an analysis of the tools that are used by students in the LMS/CMS to determine the value of the tools and how instructors can use the tools for teaching.

Article: Informing Pedagogical Action: Aligning Learning Analytics With Learning Design
Learning analytics can be aligned with learning design to inform learner intervention and course design. Learning design is intended to be used with Case-Based Learning Design. Learners participate in a case-based activity that includes a case analysis task, project task, and reflection task. Data is collected from each stage of the activity to measure learners’ engagement with activities (checkpoint analysis) and information processing (process analysis). When learners are not engaged in activities, such as accessing the content or submitting assignments, the teacher can intervene by nudging or communicating with the learners. Network diagrams are created to analyze how learners process information from the case-based learning activity. The learning analytics, teacher notes, and learner surveys (Lockyer, Heathcote, & Dawson, 2013, p. 1454) can be used to inform the redesigning of course to provide a more effective activity.

Article: Awareness is Not Enough. Pitfalls of Learning Analytics Dashboards in the Educational Practice
This article is a literature review of 26 papers that researched and published studies on the finding of learning analytics dashboards. This article seeks to evaluate the papers to identify learning analytics dashboards that incorporate learning sciences into the designing and pedagogy of the dashboards. The evaluations of the dashboards led to the identification of several theories, models, and concepts. The dashboards were classified into clusters. The clusters included: Cognitivism, Constructivism, Humanism, descriptive models, instructional design, and psychology. The dashboards were evaluated for their goals and educational concepts that affect learners. The competencies identified were metacognitive, cognitive, behavioral, emotional, and self-regulation. The dashboards were evaluated for how pedagogical interventions were designed for student support. The dashboards were classified as the following reference frames: social, achievement, and progress. The majority of the learning analytics dashboards had the foundations of self-regulated learning theory by making learners aware of their progress in the course. Learners should be presented with possible activities to support learning. The authors concluded that the dashboards have been developed for the usage of the learning data rather than improving learning with pedagogical strategies and recommend additional studies on how learners perceive the dashboards during learning.

Justification
As more courses are delivered online and use technology, there will be a significant amount of learner data that will be generated. This data can be analyzed to support learners and inform the design of courses. A community college in Arizona developed and implemented a learning analytics tool to measure learner activity in large online classes. They were able to determine when learners were accessing the learning management system, courses, assignments, and assessments. This data is used by the teachers to develop interventions to support student learning. Having access to this data will be necessary as the modality of courses converts to in-class to online. The core and first-year courses enroll a large number of learners and are required to advance to graduate. These courses are foundational as they will determine the retention and graduation of learners (Herzog, S. 2005).

Description
The study included Rio Salado College in Tempe, Arizona, and was conducted in 2008-2009. There were 61,340 unduplicated learners in an online asynchronous course. The courses were taught with the school’s LMS platform by adjunct faculty. The teacher of the courses determined the instructional design of the courses. Rio Salado College researched data mining and predictive modeling for use in courses to identify learners that require outreach and support. These models will predict when students are at risk and provide interventions to support the success of the learners. These models will also inform the instruction design of the courses.

Case Analysis
Rio Salado College developed predictive models to “identify the factors that demonstrated significant statistical correlations with final course outcomes in online courses” (Smith, V. C., Lange, A., & Huston, D. R. 2012, p. 52). The final letter grade of “C” or higher was chosen as a successful outcome in the course. The predictive models can be utilized as an early alert system to notify when an intervention is required. There was an accurate determination on when, in a course, the predictions were possible. When developing the model, the predictions would need to be real-time or near-real-time as the grades and activity of the learners frequently change during the semester. The purpose of this research is to develop connections with online learners that need support and teachers. The research questions are to answered includes identifying the factors that are early predictors of course outcomes, using factors in the predictive models to understand learner outcomes, and implementing the models to provide learner interventions and improve student success.

The online courses at Rio Salado College have a high variance in content, grades, difficulty, and learners. An online freshman-level accounting course was selected to simplify the study. There 539 learners that participated in the study during the summer and fall 2009 semesters. The study took into account the length variations of the summer semester to the fall semester. The learning management system provided student activity and assignment data. The college’s student information system provided the enrollment and final grades.

Table 1. Examples of the student activities logged in Rio Salado College’s LMS. Reprint from Predictive Modeling to Forecast Student Outcomes and Drive Effective Interventions in Online Community College Courses (p. 53), by Journal of Asynchronous Learning Networks, 16(3), 2012.
Copyright 2012 by Journal of Asynchronous Learning Networks

The predictive model used the naïve Bayes classification method due to the method being accurate, robust, and efficient. The decision tree model and the nearest neighbor model were considered and tested but were not selected. The model used warnings indicated with three levels (Low, Moderate, and High) to identify the learner’s probability of success. The Low level is above 30% of an estimated probability of success, Moderate is between 30% and 70%, and High is below 30%. The model also included activity performance metrics that displayed data to the instructors for at-risk learners to understand how to develop custom learner interventions.

Rio Salado College started with an at-risk model that ran only on the eighth day of the class and included about thirty input factors. This model was intended to be processed and determine the learner’s warning level of Low, Moderate, and High. In 2010, a new system was developed to continue to run the analysis weekly and after the eighth day of class using student performance and LMS engagement data. This system was intended to be automated and provide teachers with early alerts of at-risk learners. The teachers can use the data to develop interventions for learners that are struggling and modify course content when needed. This model seemed to produce the results that were expected to indicate the appropriate warning levels as the class progressed. The course outcome of the learners compared to the weekly warning levels. The authors stated that the “model accurately predicted the likelihood of course success at every point throughout the course” (Smith, V. C., Lange, A., & Huston, D. R. 2012, p. 58).

The initial intervention during the eight-day at-risk model was to concentrate on the learners that were at the Moderate level and contact them via telephone. These interventions were developed by a faculty group and did not improve student success rates. It is possible that there was a challenge with reaching learners directly. There is evidence of the students that received direct contact were more successful in the class than learners that were communicated to non-directly or had no communication. The predictive model determined that when learners log in to the class early, they are more successful long-term. With this data, an automated welcome email was piloted, where emails were delivered to learners before the class start date to encourage them to log in and begin engaging in the class. This intervention reduced the drop rate of learners in the pilot, but not when it was scaled in other classes.

Summary
This case study illustrates the possibilities of the impact learning analytics can have in instructional design and technology. This case study also displays the growth that learning analytics continues to have in education. While there is a massive amount of learning data to analysis, there remain to be opportunities to research new pedagogical concepts and strategies with learning analytics. There has been tremendous progress and success with learning analytics in higher education, particularly at the institutional and student success level. This is evident with the predictive analytics model developed and used as Rio Salado College and other higher education institutions. Allowing instructors and instructional designers access to the learning analytics systems can lead to improvements in course design, effective pedagogical strategies, and targeted interventions. As learning analytics studies are developed, including instructional designers in the strategizing and planning phase can provide insights on effective course design that will lead to the ideal learner outcomes. As the concepts in learning analytics mature, the understanding of how to use the analysis in pedagogy will develop. Learning analytics is a young and growing concept in learning design and technology. There remain more discoveries to be made through studies and research.

References

Baepler, P., & Murdoch, C. (2010). Academic Analytics and Data Mining in Higher Education. International Journal for the Scholarship of Teaching and Learning, 4(2). doi: 10.20429/ijsotl.2010.040217

Baker, R. S.J.d., & Yacef, K. (2009). The State of Educational Data Mining in 2009: A Review and Future Visions (Version 1.0.0). Journal of Educational Data Mining, 1(1), 3–17. doi: 10.5281/zenodo.3554658

Bienkowski, M., Feng, M. & Means, B. (2012). Enhancing Teaching and Learning through Educational Data Mining and Learning Analytics: An Issue Brief. U.S. Department of Education, 2012.

Herzog, S. (2005). Measuring Determinants of Student Return vs. Dropout/Stopout vs. Transfer: A First-to-Second Year Analysis of New Freshmen. Research in Higher Education. 46: 883. doi: 10.1007/s11162-005-6933-7

Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness Is Not Enough: Pitfalls of Learning Analytics Dashboards in the Educational Practice. Data Driven Approaches in Digital Education Lecture Notes in Computer Science, 82–96. doi: 10.1007/978-3-319-66610-5_7

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing Pedagogical Action: Aligning Learning Analytics With Learning Design. American Behavioral Scientist, 57(10), 1439–1459. doi: 10.1177/0002764213479367

Siemens, G. (2013). Learning Analytics: The Emergence of a Discipline. American Behavioral Scientist, 57(10), 1380–1400. doi: 10.1177/0002764213498851

Siemens, G. (2010). Learning Analytics & Knowledge: February 27-March 1, 2011 in Banff, Alberta: para. 4. July 7 22, 2010. Retrieved from https://tekri.athabascau.ca/analytics/

Smith, V. C., Lange, A., & Huston, D. R. (2012). Predictive Modeling to Forecast Student Outcomes and Drive Effective Interventions in Online Community College Courses. Journal of Asynchronous Learning Networks, 16(3). doi: 10.24059/olj.v16i3.275