User Population
K-12 math students using OER math curriculum
6-12 math students using Mathia, teachers using MATHia
6-16 students using Canvas
Post-secondary students preferably using OpenStax textbooks
Post-secondary online ASU students
Research Questions
Effectiveness of student supports for math learning
Improvements to student learning based on alternative presentations of material. Also motivational and related improvements due to design, messaging, etc.
Impact of learning activities and assignment context within Canvas on student mindset and performance
Learner characteristics and their influence on behavior, performance, and psychosocial constructs
ASU L@S affords a wide range of questions regarding learning in credit-bearing courses that utilize long-term and short-term student performance data and various student demographics.
Pre-Registration/Vetting
Pre-register on OSF.io
Verify feasibility of intervention with Carnegie Learning design team and interested district, including completing pre-registration form
No formal vetting process
Pre-register on OSF.io recommended
ASU Provost’s Office
IRB requirements
Normal educational practice covered by existing ASSISTments IRB, external researchers get an IRB to receive data.
Researchers use own IRB (if needed)
Researchers use own IRB
Researchers submit to Rice IRB
ASU IRB, researchers use own IRB
Recruitment
No recruitment necessary – all users eligible. The timing of the study will depend on when the teachers assign the problems as determined by the curriculum order/time of year.
Researchers recruit school(s)/district(s) using Mathia’s customer base, and may need to initiate data-sharing agreements with these districts.
Teachers (at institutions where Terracotta has been integrated) recruit students to participate in study. In the event that the researcher is not a teacher, the researcher recruits teachers to participate.
Students opt-in, incentivized, institutional partnerships
Recruitment depends on existing data or implementing interventions/surveys.
Randomization
Student-level random assignment
Individual or group random assignment (class, teacher, school, district)
Student-level random assignment AND student/assignment-level randomization (within-subject crossovers)
Student-level random assignment
Affords randomization at individual or group level depending on research question.
Intervention
Set of student supports for one or more problems
Alternate unit of instruction/activity in Mathia. Messaging, hints, presentation and design features.
Assignments
Open-ended based on capabilities of Qualtrics
Affords randomization at individual or group level depending on research question.
Prior achievement/ demographic data
Class/group membership, school/class-level contextual data, prior ASSISTments achievement
Class/group membership, prior Mathia achievement
Existing data within Canvas course site (gradebook, activity, assignments), and any student-level data added by the Teacher
Learner characteristics collected by Kinetic across studies
Data warehouse will contain demographic, achievement, course activity data
Outcome Measures
performance on Similar-but-Not-the-Same (SNS) problems
Mathia process measures, performance, and survey measures
Canvas gradebook, activity, assignments, data added by teacher
Researcher-administered measures in Qualtrics In future versions, connections to institutional data (course grades, etc)
Course activity, grades, persistence/graduation
Analysis
Data export, posted to OSF.io
Data export
Data export, possible analysis tools
Secure data enclave allows researchers to run analysis with full dataset without access to PII
Data warehouse
