Eval. of Effects of Compass Learning Odyssey Math on 4th Graders Math Achievement

Regional need and study purpose Regional Educational Laboratory (REL) Mid-Atlantic conducted a regional needs assessment that revealed student math achievement as a high priority. This led to a review of 14 studies on Odyssey® Math to understand previous evidence supporting the intervention. Seven studies reported on grade 4 students1. (See the reference section for a list of these studies.) None used random assignment to create comparable groups at baseline, so other factors that might account for group differences in outcomes were not controlled for. Clariana (2007) reports that Odyssey® Math improves student achievement in math. But this study too lacks adequate controls to fully disentangle the effect of the software from other factors. Because previous research has not controlled for confounding factors when estimating impacts of Odyssey® Math on student achievement, it is difficult to determine whether it does indeed improve student achievement.

Using a more rigorous research design than previous studies of Odyssey® Math, REL Mid-Atlantic implemented a randomized field trial to evaluate the impact of Odyssey® Math on math achievement in grade 4 in the 2007/08 school year. The study addresses one confirmatory and two exploratory questions. First, the confirmatory:

Do students in grade 4 classrooms using Odyssey® Math as a partial substitute for the regular math curriculum perform better on the math subtest of the TerraNova Basic Battery than students in control classrooms using only the regular curriculum?

The exploratory questions are: What is the effect of Odyssey® Math on the math performance gap between male and female students? What is the effect of Odyssey® Math on the math performance gap between low- and medium- or high-scoring students on a math pretest?2 While the answer to the first question can be used to inform curricular decisions, the answers to the other two questions can be used only to inform future research—as the exploratory analyses are not designed to determine whether the observed effects of Odyssey® Math (if any) are real or due to chance.3 Finally, as the first large-scale trial of Odyssey® Math, this study can inform the design of future research on the intervention in recruiting, planning power analysis, structuring implementation fidelity, and using multilevel models to analyze results.