Please ensure Javascript is enabled for purposes of website accessibility
Hamburger Nav

Research Questions

UpGrade is an open-source A/B testing platform that facilitates randomized experiments on digital learning experiences. Currently it allows experiments to contrast the type/sequence/timing of secondary math content within MATHia, but by the end of the grant period it will enable connections to other software applications.

Example: A researcher is interested in student conceptions of negative numbers. They design an alternative knowledge-building activity within the existing MATHia workspace and test whether it improves student mastery versus the default number line activity.

User Population

6-12 Mathematics students using Mathia/Teachers using MATHia

Pre-Registration/Vetting

UpGrade has previously established a process that will be a model for vetting studies during this project. This will evaluate whether the design satisfies the researchers and is acceptable to teachers and students and that it is feasible to implement within the allocated budget (both within MATHia and within UpGrade). “Design thinking” sessions, facilitated by Carnegie Learning staff, are attended by both researchers and teachers and result in a set of design prototypes for revisions to the existing workspace, which are translated into a storyboard by one of Carnegie Learning’s designers. Carnegie Learning technical staff attend the design sessions and review storyboards in order to ensure that the designs were able to be implemented within our budget. Designs were then prototyped and went through an iterative user testing process to determine usability. Continued consulting with the participants in the design sessions ensured that the final designs were consistent with their intentions. Finally, our quality assurance team tested the designs before our integration team incorporated the code into the fielded version of MATHia. For experiments run on a broad range of school districts, we have budgeted for a 10-member “educator panel,” a group of teachers and administrators representing a wide range of school districts who are familiar with MATHia and who can advise us and external researchers during this design process.

IRB requirements

Study designs should have appropriate IRB approvals from the researcher’s institution.

Recruitment (Students)

UpGrade envisions that some experiments may be exempt from IRB review or not require informed consent and could apply broadly to all students using MATHia (covered under normal educational practice by the existing Mathia IRB approval) and that others would be limited to certain students or schools. UpGrade has the ability to include or exclude particular schools and students from experiments, supporting both opt-in and opt-out scenarios. For experiments run on a broad range of school districts, UpGrade has planned a 10-member “educator panel,” a group of teachers and administrators representing a wide range of school districts who are familiar with MATHia and who can advise them and external researchers during this design process. UpGrade developers are creating a segmentation feature that will enable experimenters to easily create include/exclude rules for populations based on predefined characteristics (for instance, region, grade level, district, school-level demographics) to narrow the focus of experiments to desired targets.

Randomization

UpGrade enables experiments with random assignment at the individual, class, teacher or school level. For instance, UpGrade allows teachers or researchers to choose whether all students within a particular group membership should be randomized to the same condition (e.g. classes within teachers or teachers within schools), but could also have rules for what happens when a student transfers classes between conditions, either to maintain consistency or exclude the student. It also allows rules to be set for the individual timing and sequence of material in an adaptive software setting like MATHia, where students might see content at different times during the school year.

Intervention

UpGrade is an open-source A/B testing platform that facilitates randomized experiments on digital learning experiences. Currently it allows experiments to contrast the type/sequence/timing of secondary math content within MATHia, but by the end of the grant period it will enable connections to other software applications.

Example: A researcher is interested in student conceptions of negative numbers. They design an alternative knowledge-building activity within the existing MATHia workspace and test whether it improves student mastery versus the default number line activity.

See a demo of setting up an experiment in UpGrade here

Prior achievement/demographic data 

In the current MATHia implementation, UpGrade collects data at the student level, including all student attempts at completing a step in each problem. These data include a timestamp, the student entry, skills associated with that step, whether the student entry was considered correct, and whether it was recognized as a common error resulting in a “just-in-time” message being presented to the student. Student requests for hints are logged in the same way, as are other actions not directly related to problem solving (like viewing the glossary). As part of the overall development plan for UpGrade, the platform will include APIs and plug-ins to pass data in a flexible format back and forth from software applications other than Mathia.

MATHia allows school districts to enter unique identifiers for students, which can be used to link demographic data and pretest data (if available) with MATHia usage and performance characteristics for analysis. This is up to districts, but most (all?) districts using rostering services like Clever or OneRoster provide these IDs. Planned improvements to UpGrade will include supporting randomization that takes demographics into account (stratified random sampling). MATHia does not support use of demographic information as part of the instruction (e.g. a researcher cannot have different instructional approaches for students with different demographic characteristics).

Outcome Measures

Since MATHia assesses students as they learn, researchers can use performance in MATHia as a measure of achievement (either prior to an experiment or on a whole-year basis). This includes an aggregate measure called APLSE (Adaptive Personalized Learning Score) that acts as a measure of student achievement within MATHia and that correlates with student performance on external measures, including MAP and various state assessments. Development will also focus on authoring tools and integrations for simplifying the process of inserting external measures (e.g., survey items) into MATHia.

Analysis

UpGrade does not have support for in-platform data analysis, However, basic statistical functions (mean, median, etc) for target data collected by the client application are available for researchers to monitor some metrics of interest during data collection. Experiment parameter data can be exported for additional analyses in the researcher’s statistical package of choice.