Overview

Kajitory is a student-facing learning dashboard for an online college-level statistics course designed to support students’ understanding of their learning progress data in a “contextualized” way. Rather than using competency-based indicators, such as names of skills and learning objectives, Kajitory uses concept-based indicators, including course topic names, to communicate learning progress with students. The dashboard has shown positive preliminary data, which implies an important difference between how educators think students understand their learning and how students actually try to understand their own learning.

Whom I worked with:

Candace Thille (Stanford Graduate School of Education/Amazon) and Iris Howley (Stanford Graduate School of Education)

Methods used: 

interview, survey, paper, and digital prototyping, and storyboards

Problem:

The project was concerned about a lack of effectively-designed student-facing learning dashboards in the fields of learning analytics and the learning sciences. Despite a growing number of studies on instructor-facing learning dashboards that report student learning, there were only a few studies focused on student-facing learning dashboards a the time the project started. Existing dashboards were often a mere replication of instructor-facing dashboards, ignoring a potential difference between how educators think students learn and how students think they learn.

Approach:

Through a series of unstructured interviews with online learners, we found that many students have a hard time making sense of the given information on a learning dashboard, which is typically communicated in a form of competency-based indicators (skills and learning objectives). We addressed this issue through a lens of a categorization theory in cognitive science known as Basic Level Categories (Rosch, et al., 1976), hypothesizing that the students’ struggle comes from the mismatch between the levels at which how existing learning dashboards report student learning and how students actually try to make sense of their learning.

With the aim of creating a dashboard that communicates with students in a way that makes sense, we designed and iteratively improved several prototypes, each showing student’s learning progress using different forms of the indicator.

User test and results:

We conducted a survey (using Qualtrics) with 15 participants who had used online learning service before. When given two dashboard prototypes, one with showing student learning using a concept-based indicator (course topics) and the other using a competency-based indicator (learning objectives), more than 90% of the participants preferred the former prototype. A follow-up semi-structured interview with 8 participants revealed that students tend to interpret the information better when the information is organized using the concept-based indicator than with the competency-based indicator, primarily because the former matches the way students organize learned contents. Results also indicated that students think that competency-based indicators are meant to be used by educators and not appropriate for students.

My role included both research and design, from need-finding and prototyping to user-testing and analyzing data.