UpGrade is an open-source A/B testing platform that facilitates randomized experiments within educational software. We have integrated UpGrade with MATHia, our adaptive software used in middle and high schools across the country. Researchers can design and deploy educational experiments in UpGrade’s UI, then monitor learning outcomes and participant enrollment through a web-based dashboard. Experiments take place as part of students’ normal math instruction and take the form of alternative instructional approaches (which can include changes to content, images, videos, manipulatives, etc.). MATHia is mastery based, so existing mastery “workspaces” (aka math topics) can act as assessments as well as instruction.
Example: A researcher is interested in student conceptions of negative numbers. They design an alternative knowledge-building activity within the existing MATHia workspace and test whether it improves student mastery in the subsequent workspace.
Grades 6-12 Mathematics students using Mathia, and teachers using MATHia as part of Carnegie Learning’s blended curriculum. Carnegie Learning’s customer base includes students from over 2000 schools across the United States, reaching a broad and diverse set of student populations.
UpGrade has previously established a process that will be a model for vetting studies during this project. This will evaluate whether the design satisfies the researchers and is acceptable to teachers and students, as well as determine whether the proposed design is feasible to implement within the allocated budget (both within MATHia and within UpGrade). In some cases, “design thinking” sessions, facilitated by Carnegie Learning staff and attended by both researchers and teachers, can result in co-designed prototypes and/or storyboards for revisions to existing workspace. Carnegie Learning technical staff will review such output from the co-design sessions in order to ensure that the designs were able to be implemented within the budget. If so, designs would be prototyped, usability-tested if necessary, then and deployed in MATHia via UpGrade. Continued consulting with the participants in the co-design sessions will ensure that the final designs are consistent with their intentions. For experiments conducted within a broad range of school districts, we have budgeted for a 10-member “educator panel,” a group of teachers and administrators representing a wide range of school districts who are familiar with MATHia and who can advise Carnegie Learning staff and external researchers during this design process.
Study designs should have appropriate IRB approvals from the researcher’s institution.
UpGrade envisions that some experiments may be exempt from IRB review or not require informed consent and could apply broadly to all students using MATHia (covered under normal educational practice by the existing MATHia IRB approval) and that others would be limited to certain students or schools. UpGrade has the ability to include or exclude particular schools and students from experiments, supporting both opt-in and opt-out scenarios.
UpGrade enables experiments with random assignment at the individual, class, teacher or school level. For instance, UpGrade allows researchers to choose whether all students within a particular group should be randomized to the same condition (e.g. classes within teachers or teachers within schools), but could also have rules for what happens when a student transfers classes between conditions, either to maintain consistency or exclude the student. Student progress through MATHia is self-paced, so for experiments on specific math content, students will experience the experimental (or control) condition(s) when they reach the appropriate topic within their existing curriculum. This will typically happen at different times for different students.
UpGrade is an open-source A/B testing platform that facilitates randomized experiments on digital learning experiences. It allows researchers wide latitude to vary instructional, motivational, and/or other aspects of the software within such experiments, either generally or with respect to specific math topics.
Example: A researcher is interested in whether personalization of word problems results in a greater sense of belonging for students. The intervention inserts a belonging survey (as both pre- and post-test) in the student’s math sequence and personalizes problems to include the names of the student’s classmates as characters in the word problems.
MATHia allows school districts to enter unique identifiers for students, which can be used to link demographic data and pretest data (if available) with MATHia usage and performance characteristics for analysis. This is up to districts, but most districts use rostering vendors like Clever or ClassLink to provide these IDs. Planned improvements to UpGrade will include supporting randomization that takes demographics into account when distributing condition assignments (stratified random sampling). MATHia does not support use of demographic information as part of the instruction (e.g. a researcher cannot have different instructional approaches for students with different demographic characteristics). Researchers who wish to use demographic data would need to establish a data-sharing agreement with the school district with which they plan to work.
Find out more about the structure of Mathia data here and look at example data sets from prior studies here.
MATHia collects data at the student level, including all student attempts at completing a step in each problem. These data include a timestamp, the student entry, skills associated with that step, whether the student entry was considered correct, and whether it was recognized as a common error resulting in a “just-in-time” message being presented to the student. Student requests for hints are logged in the same way, as are other actions not directly related to problem solving (like viewing the glossary).
Since MATHia assesses students as they learn, researchers can use performance in MATHia as a measure of achievement (either prior to an experiment or on a whole-year basis). This includes an aggregate measure called APLSE (Adaptive Personalized Learning Score) that acts as a measure of student achievement within MATHia and that correlates with student performance on external measures, including MAP and various state assessments.
UpGrade allows researchers to export condition assignments and data monitored by UpGrade.
Basic statistical functions (mean, median, etc) for target data instrumented in MATHia are available for researchers to monitor metrics of interest during data collection, via UpGrade’s web-based dashboard. Condition assignments and experiment parameter data can be exported for additional analyses in the researcher’s statistical package of choice.
This project is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305N210034 to Digital Promise. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.