Challenge #1: A lot of student learning does not occur in the LMS.
Despite being called a learning management system (LMS), it is much more of a management system than a learning system. The majority of student learning does not happen within the LMS. Usually there are external resources for students to supplement their course materials. However, student interaction data in these additional materials are not being tracked. In addition, if students complete assignments in a course, it is likely that they are not working on them within the LMS. Instead, they use the LMS as an assignment submission tool. LTI and xAPI enabled tools can help solve this problem and is part of what we have implemented with our learning analytics system.
Challenge #2: LMS do not track how students interact with course materials
LMS have some analytics built-in. Most LMS will track page views, assignment submissions, grades, discussion posts/replies, and quiz scores. However, if you want information on which student has watched a certain video, or the confidence level of a student answer on an online assessment, or how much of the content page a student viewed you will need a different analytics solution.
Challenge #3: LMS do not give you real-time access to student data
In order to develop a real-time student dashboard, it is important to have real-time access to student analytics data. Most LMS have some sort of Application Programming Interface (API) to extract student data. However, these have rate limits that make pulling and analyzing data in real-time less feasible. For example, if you have 60,000 students at a university, it may take 1 week to pull all of the necessary data from the API. Now your data is 1 week old and you have to start all over again. This is not a good solution. In addition, even on a small scale, using APIs to extract data for real-time data analysis and real-time student dashboards might still take a few minutes for a class, which is longer than what is required for a real-time student dashboard.
Our solution
Create a learning analytics system that will track, store, retrieve, and display student click-level events in real-time. This will allow us to:
- Capture learning occurring in external tools outside of the LMS (LTI, xAPI enabled tools)
- Track how students are interacting with course materials at the click-level
- Have real-time access to student data in order to have a real-time student dashboard
Progress
We are running a pilot study using this analytics system in a general chemistry course right now. Students are taking quizzes in our version of Open Embedded Assessments (OEA), watching videos in a video tool called Ayamel, and interacting with our dashboards. There are about 250 students in the course, and so far we have over half a million online student interactions. The purpose of this research is to examine the effect of real-time student dashboards on student behavior and student achievement.
In my next post I will describe the design and functionality of our student dashboards: the Content Recommender dashboard and the Student Skills dashboard. I might include demo links as well so you can check them out.
