Terracotta is an open-source research platform that facilitates randomized experiments on learning activities within Canvas. It allows researchers to evaluate the content, context, timing, and mode of learning activities. The learning activities manipulated in Terracotta have the potential to be designed and implemented differentially at the class level.
Primary, secondary, and postsecondary teachers who use Canvas (and researchers who collaborate with such teachers) working at an institution that has opted to integrate Terracotta into its Canvas instance.
No formal vetting process – researchers must partner with an institution using Canvas and seek appropriate permissions to install the Terracotta plug-in. In later development, Terracotta will be integrated with OSF to enable streamlined preregistration from within Terracotta. Data exported from Terracotta is deidentified and filters non-consenting participants (if set up to do so), so there are no restrictions on data sharing.
Researchers use their own IRB. In the event of multi-institutional research, the lead researcher hosts the IRB protocol, and participating institutions are data collection sites under that single IRB. In the event of multiple researchers at different institutions, researchers should follow the guidelines of the funders or the lead PI’s institution.
The target user for Terracotta is a teacher. The teacher sets up an experiment in their class site, and recruits students to participate, effectively by assigning students to (a) submit work for an assignment that has been experimentally manipulated, or (b) submit an informed consent response providing permission from the student to participate. Researchers (who are not teachers) can be added to a Canvas site as a Teacher, which would enable the researcher to create the experiment on the teacher’s behalf. A note on privacy – Terracotta enables privacy protections for student participants, such as informed consent that is hidden from the teacher, filtering of non-consenting participants from result summaries and data exports, and removal of student identifiers from these exports. In the case of multisite research, where an experiment is being deployed across districts or institutions, Terracotta will export de-identified research data that can be shared publicly across institutional boundaries, preventing the need for complex multisite protocols.
Canvas+Terracotta will support simple A/B tests, but it will also support within-subject crossover designs (AB/BA) with pretest/posttest. This automates the difficulty of counterbalancing across experimental treatments, and also resolves concerns about bias due to experimental treatment (in within-subject crossovers, all students get all treatments, just in different orders). Moreover, Terracotta will flexibly support multiple assignments in each exposure period (AABB/BBAA) and more than two treatment conditions (ABC/CBA/BCA, where crossovers are randomized). Teachers can determine whether (and how) experimentally-manipulated assignments contribute weight to course scores.
Terracotta is an open-source A/B testing platform that facilitates randomized experiments on learning activities within Canvas. It allows researchers to evaluate the content, mode, timing, and context of learning activities. An “assignment” in Canvas can be remarkably broad, and could include instructions to do things outside the LMS, could also involve video exposure, could be a vehicle for mindset interventions, etc.
Terracotta collects item-level data about students’ responses to questions, timestamped clickstream events of students’ activity in Terracotta, grades on activities, and a complete set of contextual data about each experiment embedded in the platform. In this regard, Terracotta data are multilevel: student-level data and class-level data (about the experiment context).
There is no built-in way to link student data from Terracotta to outside data sources, and the system does not have access to demographic data outside of input provided by the instructor. Terracotta does have access to Canvas-internal identifiers, so it may be technically possible to link to other sources if appropriate data-sharing agreements were in place.
Learn more about the structure of data exports here.
Terracotta allows a researcher or teacher to collect outcome measures from the Canvas gradebook (from the LMS course site where the experiment is running), or teachers/researchers could manually enter outcome scores. By centering in the LMS, the Canvas+Terracotta platform has the ability to integrate with a class’s gradebook: any gradebook item can be selected as a research outcome (or as a pretest measure). These could be existing scores on target assignments, quizzes, or exams, and if the target outcome score is not already in the gradebook (such as statewide leveling assessments), the platform will allow manual entry or CSV upload. A researcher could also implement a custom posttest assignment within Terracotta, for example, if the outcome were measured by survey responses or student work artifacts.
Terracotta does not currently support analysis within the platform, but they plan to build capabilities to summarize data, contrast between conditions, and display summary results. This is not intended to replace the export of raw data at the conclusion of an experiment but rather to summarize experimental contrasts for monitoring and consistency. For an example analysis for a simulated experiment in Terracotta (a preregistered hierarchical Bayesian model), see https://osf.io/jwk4z/
This project is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305N210034 to Digital Promise. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.