UpGrade is an open-source A/B testing platform that facilitates randomized experiments within MATHia, our adaptive software used in middle and high schools across the country. Experiments take place as part of students’ normal math instruction and take the form of alternative instructional approaches (which can include changes to content, images, videos, manipulatives, etc.). MATHia is mastery based, so existing mastery “workspaces” can act as assessments as well as instruction.
Example: A researcher is interested in student conceptions of negative numbers. They design an alternative knowledge-building activity within the existing MATHia workspace and test whether it improves student mastery in the subsequent workspace.
6-12 Mathematics students using Mathia/Teachers using MATHia
UpGrade has previously established a process that will be a model for vetting studies during this project. This will evaluate whether the design satisfies the researchers and is acceptable to teachers and students and that it is feasible to implement within the allocated budget (both within MATHia and within UpGrade). In some cases, “design thinking” sessions, facilitated by Carnegie Learning staff, are attended by both researchers and teachers and result in a set of design prototypes for revisions to the existing workspace, which are translated into a storyboard by one of Carnegie Learning’s designers. Carnegie Learning technical staff attend the design sessions and review storyboards in order to ensure that the designs were able to be implemented within our budget. Designs were then prototyped and went through an iterative user testing process to determine usability. Continued consulting with the participants in the design sessions ensured that the final designs were consistent with their intentions. Finally, our quality assurance team tested the designs before our integration team incorporated the code into the fielded version of MATHia. For experiments run on a broad range of school districts, we have budgeted for a 10-member “educator panel,” a group of teachers and administrators representing a wide range of school districts who are familiar with MATHia and who can advise us and external researchers during this design process.
Study designs should have appropriate IRB approvals from the researcher’s institution.
UpGrade envisions that some experiments may be exempt from IRB review or not require informed consent and could apply broadly to all students using MATHia (covered under normal educational practice by the existing Mathia IRB approval) and that others would be limited to certain students or schools. UpGrade has the ability to include or exclude particular schools and students from experiments, supporting both opt-in and opt-out scenarios.
UpGrade enables experiments with random assignment at the individual, class, teacher or school level. For instance, UpGrade allows teachers or researchers to choose whether all students within a particular group should be randomized to the same condition (e.g. classes within teachers or teachers within schools), but could also have rules for what happens when a student transfers classes between conditions, either to maintain consistency or exclude the student. Student progress through MATHia is self-paced so, for experiments on specific math content, students will experience the experimental (or control) condition(s) when they get to the appropriate topic. This will typically happen at different times for different students.
UpGrade is an open-source A/B testing platform that facilitates randomized experiments on digital learning experiences. It allows experiments wide latitude to vary instructional and motivational aspects of the software, either generally or with respect to specific math topics.
Example: A researcher is interested in whether personalization of word problems results in a greater sense of belonging for students. The intervention inserts a belonging survey (as both pre- and post-test) in the student’s math sequence and personalizes problems to include the names of the student’s classmates as characters in the word problems.
MATHia allows school districts to enter unique identifiers for students, which can be used to link demographic data and pretest data (if available) with MATHia usage and performance characteristics for analysis. This is up to districts, but most (all?) districts using rostering services like Clever or OneRoster provide these IDs. Planned improvements to UpGrade will include supporting randomization that takes demographics into account (stratified random sampling). MATHia does not support use of demographic information as part of the instruction (e.g. a researcher cannot have different instructional approaches for students with different demographic characteristics).
Find out more about the structure of Mathia data here and look at example data sets from prior studies here.
MATHia collects data at the student level, including all student attempts at completing a step in each problem. These data include a timestamp, the student entry, skills associated with that step, whether the student entry was considered correct, and whether it was recognized as a common error resulting in a “just-in-time” message being presented to the student. Student requests for hints are logged in the same way, as are other actions not directly related to problem solving (like viewing the glossary).
Since MATHia assesses students as they learn, researchers can use performance in MATHia as a measure of achievement (either prior to an experiment or on a whole-year basis). This includes an aggregate measure called APLSE (Adaptive Personalized Learning Score) that acts as a measure of student achievement within MATHia and that correlates with student performance on external measures, including MAP and various state assessments.
UpGrade allows researchers to export condition assignments and data monitored by UpGrade.
Basic statistical functions (mean, median, etc) for target data collected by MATHia are available for researchers to monitor metrics of interest during data collection. Condition assignments and experiment parameter data can be exported for additional analyses in the researcher’s statistical package of choice.
This project is supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305N210034 to Digital Promise. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.