Compared to reading, I think math is a better reflection of what a kid learns in school. Reading is more tied to vocabulary and parental SES.
I'm somewhat indifferent on the grade levels, but I went with 8th here for two reasons. One, it should show more about what value schools are adding on top of any incoming knowledge. And two, there are high-quality research studies linking 8th grade math performance to longer-term economic outcomes. I suspect the same findings might hold true for 4th grade math as well, but that research has been done on 8th grade.
8th grade is considered a “gateway “ point in K-12. This article examines absolute performance and then handicaps with other weighted attributes . Not sure I would recommend moving from Mass based on one metric alone. Mass is still the best option all things considered.
There isn’t a deep dive into the curriculum, methods, or preparation in the other grades leading up to 8th.
For an educator “growth” is the best indicator of the “herd” which can validate and lead to improvement with caveats.
A .225 hitter who raises their average to .250 is cause for celebration. However, if the absolute “average” for their position is .260 then it can be determined to be under performing to the standard. Who do want in the lineup?
As a parent it can be perplexing in looking at educational statistics because of the multiple variables involved. That said if Mass is leading perennially in the standings it is fairly safe to say they are controlling for most of the variables. Go Sox!
You say that UI controlled for demographics, but then that you took it a step forward by controlling for demographics. What’s the difference btwn what UI did and what you did?
“The Urban Institute’s Matt Chingos recently ran the numbers for every state. He controlled for student demographics including gender, age, race or ethnicity, receipt of free and reduced-price lunch, special education status, and English language learner status, and then adjusted the scores accordingly. Across 4th and 8th grade math and reading Massachusetts still came in first among all the states
But I took it one step farther to find out which states did better or worse after controlling for demographics. “
Oh, sorry if that was confusing. I just looked to see how much UI's demographically-controlled scores differed from the state's *actual* scores. I only did it for 8th grade math, but their report has the scores for 4th and 8th grade reading and math.
I’m still trying to figure out the upshot. Is this right?
1. No parents should select a certain state’s schools based on unweighted test results, since these are mostly compositional effects.
2. UI acknowledges that, and controls for compositional effects to try to determine actual school performance. Their implication is that MA schools, for the average kid, perform best.
3. You agree with that, but tease out differences in racial demographics (as an example.) Your finding is that MA loses a lot more performance in this measure, but is presumably still #1 overall because it does so well with white kids (even relative to white kids in other states.)
So as a parent of a white kid, I can assume MA is the best school system for my kid (based on performance for that demo, not compositional effects.) But as a parent of a black kid, I can assume other states are better, because controlling for demographic hurt MA and helped other states.
Well, it's the difference between looking at a state's schools versus its scores. Aka, does it have good schools, or do they just have a lot of wealthy families who value education? This measure is saying that Massachusetts has excellent scores but just middling schools.
The UI analysis looks at all demographic factors together and doesn't split out the results by race. But you could just look at the raw NAEP results by race/ ethnicity and see that MA does pretty well for White and Black students but very poorly for its English Learners.
As a retired Principal in a small town with low socio-economic demographics in West Texas, I have been frustrated for YEARS that we do a BETTER job educating our kids that most "Rich and White" school districts in Texas and get no credit because of "who" attends those schools and what kind of education they receive at ours. This article articulates and shows EXACTLY what education means! THANK YOU, and I will spread this information and Mr. Aldeman's Substack account all I can!
The major flaw in this is NM students still have to be able to compete on a global level. This chart makes it look like they would be able to do that well. No. The low scores affect remediation, completion and success in college and career. Students cannot read technical manuals in a blue collar job or do basic algebra as a freshman in college. The best schools in NM get "Massachusets" results despite hardships and demographics. It is happening in certain schools across NM, so it is possible. That is true equity. I worry about glorifying poor results due to demographics, it doesn't help real people in their lives.
Huh? The post links to the Urban Institute report and says it controls for "student demographics including gender, age, race or ethnicity, receipt of free and reduced-price lunch, special education status, and English language learner status."
Interesting stuff. I think another take may be “look what education does to demographics.” In the sense that the overperforming states (NM, SC, MS) are historically poorer, and many of these are southern states or with high spanish-speaking demographics. There’s a lot of catch up growth in the south, not just economically, but culturally, and in education over the last half centurty. I think the deep south didn’t reach 80% of the countries median income until like the mid-80s. There was so much economic/academic potential lost in these states due to poverty and segregation. “overperforming your demographics” could just be another way of saying “we have historically disadvantaged populations that are benefiting from a functioning, inclusive educational institutions that wasn’t available to other states 2 generations ago” and underperforming could be “we don’t have those communities.” Just my two pennies worth!
Am I right to summarize this finding as, "the most advantaged students (white, more wealthy) in Massachusetts outperform demographically similar students in other states, while less advantaged students (Black and Latino, less wealthy) underperform demographically similar students in other states?
No, the analysis doesn't separate out the results for specific demographic groups. Instead, it's looking at how a state does given all of its demographics factors. Aka, if MA students were perfectly representative of the country writ large, its scores would be 3.5 points lower than what they actually are.
thank you for the reply and explanation!! And please no pressure to keep responding, I know you have other important things to do. It predicts a score based on demographic characteristics (ie how other demographically similar students nationwide performed) and compares individual student scores within a state to their predicted score, is that right? So while it's not making state-to-state subgroup comparisons, it is comparing demographically similar students, if I understand. So I'm wondering how MA would be ranked first on demographically adjusted score? And it seems that the explanation is that, while demographically disadvantaged students do worse (in order to bring the adjusted score down relative to other states), demographically advantaged students must do better than their peers in order to bring the adjusted average up above those states. This is purely out of curiosity as I try to understand this very interesting finding.
Hi Tony, no problem, let me try again. It's not necessarily saying anything about MA's Black or White students or any particular subgroup. Instead, it's saying that, given the full mix of MA's student body, it should be doing better.
I'm not sure if this will help, but let me try with a metaphor. Imagine a basketball team. It has 5 free-throw shooters who average 0, 0, 0, 50, and 100 percent, respectively. For a given game, the team might make 50 percent of its free-throws. But to know whether that was good or bad, you'd have to know how many shots each player took.
I am wondering if this tool is available for individual school district to analyze their metrics? Thank you!
Hi Mariah, I'm not aware of anything quite like this at the district level. You can see my attempt at something similar for math here: https://www.the74million.org/article/which-school-districts-do-the-best-job-of-teaching-math/
Or this project from Stanford attempts to standardize district performance across states: https://edopportunity.org/
Thank you! I will check it out. I appreciate your efforts in this matter.
The did you select 8th grade math for your comparison? Is there a reason you think it’s a better metric than 4-8 math and reading?
Compared to reading, I think math is a better reflection of what a kid learns in school. Reading is more tied to vocabulary and parental SES.
I'm somewhat indifferent on the grade levels, but I went with 8th here for two reasons. One, it should show more about what value schools are adding on top of any incoming knowledge. And two, there are high-quality research studies linking 8th grade math performance to longer-term economic outcomes. I suspect the same findings might hold true for 4th grade math as well, but that research has been done on 8th grade.
8th grade is considered a “gateway “ point in K-12. This article examines absolute performance and then handicaps with other weighted attributes . Not sure I would recommend moving from Mass based on one metric alone. Mass is still the best option all things considered.
There isn’t a deep dive into the curriculum, methods, or preparation in the other grades leading up to 8th.
For an educator “growth” is the best indicator of the “herd” which can validate and lead to improvement with caveats.
A .225 hitter who raises their average to .250 is cause for celebration. However, if the absolute “average” for their position is .260 then it can be determined to be under performing to the standard. Who do want in the lineup?
As a parent it can be perplexing in looking at educational statistics because of the multiple variables involved. That said if Mass is leading perennially in the standings it is fairly safe to say they are controlling for most of the variables. Go Sox!
I was wondering the same thing and this explanation makes sense.
You say that UI controlled for demographics, but then that you took it a step forward by controlling for demographics. What’s the difference btwn what UI did and what you did?
“The Urban Institute’s Matt Chingos recently ran the numbers for every state. He controlled for student demographics including gender, age, race or ethnicity, receipt of free and reduced-price lunch, special education status, and English language learner status, and then adjusted the scores accordingly. Across 4th and 8th grade math and reading Massachusetts still came in first among all the states
But I took it one step farther to find out which states did better or worse after controlling for demographics. “
Oh, sorry if that was confusing. I just looked to see how much UI's demographically-controlled scores differed from the state's *actual* scores. I only did it for 8th grade math, but their report has the scores for 4th and 8th grade reading and math.
I’m still trying to figure out the upshot. Is this right?
1. No parents should select a certain state’s schools based on unweighted test results, since these are mostly compositional effects.
2. UI acknowledges that, and controls for compositional effects to try to determine actual school performance. Their implication is that MA schools, for the average kid, perform best.
3. You agree with that, but tease out differences in racial demographics (as an example.) Your finding is that MA loses a lot more performance in this measure, but is presumably still #1 overall because it does so well with white kids (even relative to white kids in other states.)
So as a parent of a white kid, I can assume MA is the best school system for my kid (based on performance for that demo, not compositional effects.) But as a parent of a black kid, I can assume other states are better, because controlling for demographic hurt MA and helped other states.
Well, it's the difference between looking at a state's schools versus its scores. Aka, does it have good schools, or do they just have a lot of wealthy families who value education? This measure is saying that Massachusetts has excellent scores but just middling schools.
The UI analysis looks at all demographic factors together and doesn't split out the results by race. But you could just look at the raw NAEP results by race/ ethnicity and see that MA does pretty well for White and Black students but very poorly for its English Learners.
As a retired Principal in a small town with low socio-economic demographics in West Texas, I have been frustrated for YEARS that we do a BETTER job educating our kids that most "Rich and White" school districts in Texas and get no credit because of "who" attends those schools and what kind of education they receive at ours. This article articulates and shows EXACTLY what education means! THANK YOU, and I will spread this information and Mr. Aldeman's Substack account all I can!
Do you mean “corrected text as of 1/28/25” ?
Ugh, yes, thanks for flagging that.
It’s based on testing that’s a bunch of BS
This is just dumb
The major flaw in this is NM students still have to be able to compete on a global level. This chart makes it look like they would be able to do that well. No. The low scores affect remediation, completion and success in college and career. Students cannot read technical manuals in a blue collar job or do basic algebra as a freshman in college. The best schools in NM get "Massachusets" results despite hardships and demographics. It is happening in certain schools across NM, so it is possible. That is true equity. I worry about glorifying poor results due to demographics, it doesn't help real people in their lives.
David Adler from ReContext developed a tool to do this at the school level. It was available in Vermont and North Carolina.
I suppose we can only speculate what “control for demographics” even means.
Huh? The post links to the Urban Institute report and says it controls for "student demographics including gender, age, race or ethnicity, receipt of free and reduced-price lunch, special education status, and English language learner status."
Interesting stuff. I think another take may be “look what education does to demographics.” In the sense that the overperforming states (NM, SC, MS) are historically poorer, and many of these are southern states or with high spanish-speaking demographics. There’s a lot of catch up growth in the south, not just economically, but culturally, and in education over the last half centurty. I think the deep south didn’t reach 80% of the countries median income until like the mid-80s. There was so much economic/academic potential lost in these states due to poverty and segregation. “overperforming your demographics” could just be another way of saying “we have historically disadvantaged populations that are benefiting from a functioning, inclusive educational institutions that wasn’t available to other states 2 generations ago” and underperforming could be “we don’t have those communities.” Just my two pennies worth!
Am I right to summarize this finding as, "the most advantaged students (white, more wealthy) in Massachusetts outperform demographically similar students in other states, while less advantaged students (Black and Latino, less wealthy) underperform demographically similar students in other states?
No, the analysis doesn't separate out the results for specific demographic groups. Instead, it's looking at how a state does given all of its demographics factors. Aka, if MA students were perfectly representative of the country writ large, its scores would be 3.5 points lower than what they actually are.
thank you for the reply and explanation!! And please no pressure to keep responding, I know you have other important things to do. It predicts a score based on demographic characteristics (ie how other demographically similar students nationwide performed) and compares individual student scores within a state to their predicted score, is that right? So while it's not making state-to-state subgroup comparisons, it is comparing demographically similar students, if I understand. So I'm wondering how MA would be ranked first on demographically adjusted score? And it seems that the explanation is that, while demographically disadvantaged students do worse (in order to bring the adjusted score down relative to other states), demographically advantaged students must do better than their peers in order to bring the adjusted average up above those states. This is purely out of curiosity as I try to understand this very interesting finding.
Hi Tony, no problem, let me try again. It's not necessarily saying anything about MA's Black or White students or any particular subgroup. Instead, it's saying that, given the full mix of MA's student body, it should be doing better.
I'm not sure if this will help, but let me try with a metaphor. Imagine a basketball team. It has 5 free-throw shooters who average 0, 0, 0, 50, and 100 percent, respectively. For a given game, the team might make 50 percent of its free-throws. But to know whether that was good or bad, you'd have to know how many shots each player took.