Estimating ST Math Benefit

We want to estimate how much students at A California School District benefit from their use of ST Math. Do students who experience a high amount of ST Math content outperform those who experience a medium amount, or a low amount? Note that our theory of change is that for every ST Math Learning Objective students complete, solving its puzzles and passing its levels, they have demonstrated a visual understanding of that Objective’s specific math content. This then improves students’ average expected performance on any assessment items aligned to that Learning Objective. Learning Objectives are aligned to grade-level state math standards. The amount of math content experienced is indicated by the number of Learning Objectives completed, which is represented by the student’s percent “Progress” in ST Math.

The Data

The students took the math portion of the Smarter Balanced Assessment Consortium Assessment, first in spring 22 and then in spring 23. After joining the table of Smarter Balanced Assessment Consortium student scores with MIND’s table of 2022-2023 school year student ST Math usage*, we end up with a merged analytic sample of 5565 students in 3 grade cohorts (see Table 1).

Note: usage in terms of puzzles collected is determined for the time period between the initial, spring 22 SBAC testing date and the final, spring 23 SBAC testing date.

Controlling for Math Content Knowledge

Of course, the strongest predictor of where students end up on Spring 2022-2023 Scores is where they began: their Spring 2021-2022 Scores. Their Spring 2021-2022 Scale Scores are also a way to measure each students’ math content knowledge prior to the ST Math treatment. In our model we will use students’ Spring 2021-2022 Scores to represent their initial Math Content Knowledge (MCK).

In addition to predicting scale scores, prior MCK also affects the amount of ST Math Progress (see Figure 3). Students with high prior MCK use it to fail fewer puzzles and thus fail fewer levels in ST Math. Meanwhile, students with low prior MCK fail more puzzles and levels. Since every level must be passed with a score of 100% in order to move ahead, the lower MCK students have to replay more levels before ultimately passing them, thus taking more time. In other words, higher MCK students enjoy higher progress rates per minute played. Given equal minutes on ST Math, the higher MCK students could end up over-represented at high ST Math Progress values.

For both of these reasons, it is crucial to factor prior MCK alongside Content Progress into the hierarchical linear models predicting SBAC scale scores. Whatever SBAC scale score growth MCK can not predict on its own is left to the coefficient of ST Math Progress.

The scatter plots in Figure 1 show the strong correlations between Spring 2021-2022 and Spring 2022-2023 scale scores.

Modeling Variable Relationships

In this observational study, we use hierarchical linear models to control for school and classroom (teacher) effects. In sum, our growth estimate due to ST Math content is controlled for: (i) the students’ potential to grow (based on their initial SBAC scale score), (ii) the effect of MCK on ST Math content progress (number of puzzles collected), and (iii) any fixed average differences between each of the schools and/or teachers.

The diagram (Figure 2) and three plots (Figure 3-5) help illustrate the relationships between the independent and dependent variables in the models (one model per grade). Each 2-variable plot cannot tell the full story, but they motivate the need to model the data appropriately to estimate the impact that ST Math has on growth. (See Appendix for grade breakdowns)

Results

With prior math content knowledge as well as school/teacher fixed effects factored in, there still remains an SBAC Scale Score difference between students who reached either high, medium, or low amount of ST Math Progress. The linear modeling generates an intercept and two “coefficients”: one coefficient for prior MCK (represented by Spring 2021-2022 SBAC Scale Score) and a second coefficient for ST Math Puzzles Collected.

The columns in Table 2 summarize the estimated model coefficients (slopes and intercepts) for all students combined. Note that the standard errors are shown in parenthesis.

**Table 2: *Hierarchical Regression* Model Results**

	Dependent variable:

	Spring 2021-2022 to Spring 2022-2023 SBAC Scale Score Growth
	4th Grade	5th Grade	6th Grade

ST Math Progress Coefficient	41.67^***	52.78^***	42.04^***
	(3.48)	(4.21)	(4.45)

Spring 2021-2022 SBAC Scale Score Coefficient	-0.26^***	-0.22^***	-0.13^***
	(0.02)	(0.02)	(0.02)

Intercept Value	653.17^***	529.74^***	319.12^***
	(36.25)	(40.87)	(45.54)


Observations	1,820	1,801	1,944

Statistical Significance Levels	p<0.10 significance: *
	p<0.05 significance: **
	p<0.01 significance: ***

The ST Math progress coefficient values represent the contribution that ST Math content coverage had on scale score growth from Spring 2021-2022 to Spring 2022-2023. Because these are linear models, the coefficients mean

For each 10% of ST Math progress earned between exams, a 6th grade student would be expected to grow by an additional \(\frac{42.04}{10} = 4.2\) SBAC Scale Score points.

Applying the Models to Estimate Effects of Hypothetical ST Math Usage Benchmarks for the whole district.

We can also use our models above to adjust what each student’s individual SBAC Scale Score would be at any hypothetical ST Math progress. We can thus evaluate the hypothetical scenarios where all students are modeled as reaching specific, uniform ST Math progress benchmarks whether lower or higher from their actual.

In the following section, we estimate the effects of adjusting individual student’s ST Math content progress to each one of four different benchmarks: =0, minimum 25%, minimum 50%, and minimum 75%, and minimum 100% Progress Each individual student’s SBAC Scale Score is adjusted for their net change in actual content progress to align with these benchmarks.

Below are the resulting simulated Smarter Balanced Assessment Consortium proficiency level percentage distributions by grade level corresponding to the adjusted Spring 2022-2023 SBAC Scale Scores. On the left is the actual SBAC Scale Score proficiency level distribution corresponding to A California School District’s actual SBAC scores. On the right are the estimated SBAC proficiency level distributions corresponding to hypothetical ST Math usage benchmarks of 0, \(\geq\) 25%, \(\geq\) 50%, \(\geq\) 75%, and \(\geq\) 100% content progress.

Note that this for this analysis, we use this mapping to map SBAC Scale Score to their equivalent proficiency levels.

Note: If the student belongs to :

proficiency Lo means the student’s score is between 1st and 20th percentiles;
proficiency LoAvg means the student’s score is between 21st and 40th percentiles;
proficiency Avg means the student’s score is between 41st and 60th percentiles;
proficiency HiAvg means the student’s score is between 61st and 80th percentiles;
proficiency Hi means the student’s score is above 80th percentile.

4th Grade SBAC Proficiency Levels

Note: 4th grade coefficent is not statistically significant.

5th Grade SBAC Proficiency Levels

6th Grade SBAC Proficiency Levels

Summary

Our hierarchical linear models show that higher ST Math content coverage (Progress) correlates with higher growth in SBAC Scale Score (and thus increases in SBAC proficiency levels) after controlling for students’ MCK (using students’ Spring 2021-2022 SBAC Scale Scores) and adjusting for school/teacher effects (using hierarchical regression modeling). Averaged across the grade-cohorts, with statistically significant results, we estimate that each additional 10% of ST Math Progress contributes to an additional 4.5 amount of extra points in raw SBAC scale score growth.