Representatives from the teams at UpGrade and eTrials presented at the 2022 Conference on Digital Experimentation at MIT (CoDE@MIT). One key insight from the UpGrade team was that there is no single best time for each student to participate in an experiment, so experiments should occur over time. The ASSISTments shared the availability of an open data set with 30,000 students and a surprising insight: motivational messages can have a negative impact. To learn more from each presentation, read the blog below.
Presentation By April Murphy and Steve Ritter, Carnegie Learning
The UpGrade A/B testing platform helps researchers implement large-scale, classroom-based experiments using the adaptive software MATHia. It is also designed to help researchers navigate challenges in experimental design and deployment, such as the ways in which adaptive software allows personalization of content and students to advance through content at different rates. They discuss three key design considerations for conducting digital experiments within adaptive educational software: ordering and sequencing, coordination of experimental activities, and exclusion criteria.
1. Ordering and sequencing.How can researchers conduct experiments at the best time for the students involved? Insight: There is no single best time!
Much like teachers might customize the presentation of a print textbook for their learners, adaptive software may include, omit, extend, or contract the presentation of particular topics for students based on an assessment of their needs. This means that in a widely deployed adaptive instructional software like MATHia, there is no optimal time to run an experiment for all students, but launching an experiment any time in the school year can provide useful data from some students. Researchers assessing the impact of a particular activity can conduct a “curriculum-embedded” experiment, where students encounter the experiment, automatically and without disruption, at the appropriate time in their own instructional sequence. This works particularly well for conducting experiments across many states and districts, where students’ instructional timelines may vary greatly. When the participant pool is thousands or millions of students, as it is with UpGrade, the smaller sample size may still provide more data, more quickly, and with greater statistical power than small scale approaches.
2. Coordination of experimental activities: How should randomization occur? Insight: A platform must support multiple options.
Some experiments may be designed to extend across multiple activities, such as those that add experimental interventions to multiple modules within a topic. Decisions regarding how and when randomization occurs are based on the goals of the experiment and educational content. Randomization may occur at the start of the experiment and be coordinated across activities, so that a student either receives all or none of the experimental interventions. Or, randomization may occur independently at each experimental point, with participants potentially receiving different condition assignments at each decision point. This method creates a “dosage effect” because each student would be eligible to receive the experimental condition a varied number of times. A third option is to control the dosage in a within-subjects experiment. UpGrade allows experimenters to specify how to manage coordination of experimental activities, and tries not to advise researchers on which to choose.
3. Student selection reasoning: Which students should be included in an experiment? Insight: Not only do the characteristics of students matter, but also their prior learning experiences.
In some experiments, it will be necessary to account for students’ prior learning experiences, even before the experiment begins. Experiments may want to distinguish between first and subsequent encounters of an activity, and a researcher may desire to have prior experience with an activity in order for a student to quality to be selected for the experiment.. UpGrade has implemented the ability to automatically select students for an experiment based on their prior learning experiences, and allow experimenters to determine whether, as in the “dosage effect”, a student assigned to the experimental condition must experience all possible activities associated with that condition to be included in the experiment.
To read the full paper, which includes authentic example experimental designs, visit UpGrade’s Resource Center or click here for a PDF of the paper.
Presented By Aaron Haim, Ethan Prihar, Stacy T. Shaw, Adam Sales, and Neil T. Heffernan, Worcester Polytechnic Institute
ASSISTments is leading the field with their release of publicly available, award winning datasets. By making their datasets “open data” – publicly available and able to be used without restriction – they are advancing Open Science practices and the SEER principles.
One of these datasets, a large dataset containing fifty experiments, was run by researchers in E-TRIALS. The dataset contains 50,752 instances across 30,408 students. Some of the
experiments within the dataset included fourteen experiments across 12,243 students injecting motivational messages and videos into the assignment, five experiments across 2,492 students comparing tutoring provided in a text-based versus a video format, and two experiments across 4,057 students comparing filling in the correct answer versus selecting from multiple preset options for a given problem. The experiments found that motivational messages had a negative impact on student learning, students learned more from tutoring in a video format compared to text-based, and students learned more where they filled in the correct answers compared to selecting the answer from multiple options. The dataset received the Best Publicly Available Educational Data Set award at the 15th International Conference on Educational Data Mining.
Within the field of education technology, open science practices are sparse or stagnant at times, in part due to the challenges to making education data openly available. ASSISTments only releases students’ answers to math questions so that no personally identifiable information is ever disclosed.
Resources and education on open science practices need to be made more readily available to provide researchers with the necessary understanding to make their work more openly accessible (whether through methodology, dataset, analysis, preregistration, etc.) and to
mitigate any potential issues when future researchers choose to reproduce or replicate existing
work. This work provides some starting work and examples for complying and requiring other
researchers to comply with open science practices.
To read the full paper and access datasets, visit this site.
This project is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305N210034 to Digital Promise. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.