At the core of SEERNet’s mission is the empowerment of researchers to test interventions in diverse digital environments at a larger scale. This approach aims to ensure that the interventions are responsive to the needs of learners from diverse backgrounds. To that end, our DLPs are constantly working to improve their platforms to ensure usability and accessibility for a wide-range of researchers and use scenarios. In this blog post, Ben Motz, Assistant Professor, Department of Psychological and Brain Sciences and Associate Director of the eLearning Research and Practice Lab in Indiana University’s Pervasive Technology Institute, shares recent developments within the Terracotta digital learning platform.
Terracotta is a plug-in that integrates with the learning management system (LMS) Canvas and makes it possible for teachers and researchers (in collaboration with teachers) to embed studies directly in a LMS course site. Canvas is the leading LMS in both the global and North American marketplace, used in 30% of institutions in K12 districts and universities, and by 38% of students in North America. Institutions and districts buy and manage their own Canvas instances from Instructure (the company who produces Canvas), and have control over how Canvas operates in their institutions and districts, in terms of functionality and modules included. Basically, each institution decides how it wants Canvas to work, and each class establishes its own norms within its course site.
Terracotta is founded on the belief that research should inform instructional practice, and that experiments should be based on what teachers actually do in their classrooms. Teachers and students alike spend much of their time in the learning management system. Since Terracotta integrates into Canvas, a place where teachers already are, Terracotta eliminates itself as a gatekeeper in the research process. Teachers are able to run their own experiments and don’t need to go through an intermediary. All teachers (or their institutions or districts, depending on how Canvas is set up) need to do is turn on the Terracotta integration in their course, without the Terracotta team needing to program anything. Within Terracotta, teachers can create different versions of course assignments, implement informed consent, and randomly assign students to different versions of a single assignment, or string assignments together to create a sequence of exposures to different conditions.
Since Terracotta is integrated within Canvas course sites, Terracotta doesn’t have built-in users like many digital learning platforms. Instead, external researchers form relationships with schools and teachers to deploy a study, though teachers can also choose to run their own studies. Lastly, because both Terracotta and the researchers using Terracotta have to collaborate so closely with districts and schools, Terracotta is focused on the ethical considerations of educational research. Students and teachers have the agency to opt-out of studies, providing for greater good will between researcher and study participant.
Because Terracotta’s users aren’t necessarily trained in quantitative data analysis, Terracotta needs to display results in user-friendly ways. A new functional prototype of a data dashboard is being rolled out to users. On the results dashboard, there are two tabs: the overview tab and the outcomes tab. The overview tab gives a full picture of the experiment, including participants, assignments, conditions, and experiment-type. The outcomes tab shows visualizations of a variety of different outcomes, ranging from time on task to distal outcomes, such as an exam score, that might be affected by the conditions being tested within Terracotta. Users can also download the data and visualizations.
Terracotta also recently conducted user research to better understand user behaviors, needs, and motivations. Based on survey responses from 21 past users, including both teachers and researchers, Terracotta received a system usability scale (SUS) score of 74.5, which is higher than the score for iPhones, Powerpoint, or Canvas itself. The survey questionnaire also asked Terracotta-specific questions, and found users to be wholly satisfied with their use of Terracotta. The callout below shows levels of user agreement with the following statements:
Terracotta helps me conduct research in my class: 100%
I conduct more research in my class now that I have access to Terracotta: 52%
Terracotta allows me to conduct new types of research in my classes: 84%
Terracotta helps me run more ethical experiments: 71%
Terracotta reduces challenges I have faced when conducting research in my classes: 89%
In addition to surveys, the Terracotta team has conducted detailed user interviews, where they asked users to walk through the process of completing a specific task. Based on these interviews, the team has identified a few places within the platform’s user interface where changes would smooth the way for users, like adding definitions in hover text for some technical terms, and automatically expanding assignments to show the different versions the teacher will need to create. The team is continuing to analyze the results of these user interviews in order to pull out further insights that will help improve Terracotta’s usability.
For more information about Terracotta, visit our website, and stay informed about new SEERNet updates by joining the SEERNet interest list and join the discussion on Twitter/X with #SEERNet!
This project is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305N210034 to Digital Promise. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
