The approach Colorado uses to measure, rate and bolster schools and districts is working like it’s supposed to, according to a legislative mandated audit, which said the accountability system is a “reasonable and appropriate” way to evaluate school performance.
Not everyone agrees with that assessment, though. Some Colorado superintendents said the system is inequitable, inaccurate and doesn’t effectively help school districts prepare students for the 21st century. And the majority of parents said the system doesn’t provide useful information about how their kids are doing in school.
State lawmakers passed a bill last year ordering the independent audit, which was released on Monday.
The bill asked the audit to answer several questions: Has the accountability system improved education? Is it improving student outcomes? Are students living in poverty unfairly hurt by the accountability system, and can schools be fairly compared under it? Do unintended barriers or obstacles exist that inadvertently hurt marginalized students? Is it giving educators the data they need to improve?
Auditors said the system is generally designed to do what it’s supposed to.
“We did not identify any significant gaps in the design of the accountability system,” the audit by the Human Resources Research Organization, which was contracted by the Office of the State Auditor.
The audit didn’t provide clear answers on two key questions: whether the system is biased against schools with high numbers of students living in poverty and why outcomes are what they are.
The audit examined two systems, the 2008 system of standards and assessments, and the 2009 accountability system.
What do Colorado’s school accountability and accreditation systems do?
The accountability system measures performance in schools and school districts using several measures: standardized test scores, graduation and dropout rates, and for high schools, college enrollment rates. The accreditation system rates those schools and districts and if they land in one of the two lowest-performing levels, they get placed on the state’s watch list.
Schools come under scrutiny and state education officials assist districts with improvement plans. If districts or schools don’t improve for five consecutive years, the state board can intervene: closing schools, converting them to charter schools or ordering a third party to take control.
What did the audit find?
Overall, it found that schools and districts are assigned performance ratings consistent with how well they do on each of the performance measures, for example, how much students grow academically in one year. But even that finding baffled some on the audit committee.
State Rep. Colin Larson, a member of the audit committee, asked how can 70 percent of districts be in the top two performance categories when they have 41 percent of their high school students not reading and writing at grade levels, 45 percent not on grade level at math and 53 percent not meeting science standards at grade level.
“How do we have that disconnect in an accountability system?”
It is because “achievement” on a single standardized test is only one measure in the accountability system. It also looks at graduation rates, career and college readiness, and how much students grow academically in one year.
“Even if students come in at a lower level, we tend to give them more credit for growing students,” said Katy Anthes, commissioner of education.
The audit found what decades of national education research has shown: Schools with higher proportions of students receiving free or reduced-price lunch, Hispanic or Black students, and students with disabilities generally had lower overall academic achievement. That means that schools or districts with high numbers of students of color are more likely to receive a low performance rating.
The audit didn’t make a judgment on whether that unfairly punishes those districts or hurts students.
Arthur Thacker of Human Resources Research Organization said interviews with district administrators revealed that many viewed low performance ratings as punitive “and that created challenges for staff recruitment and for keeping students which could hinder some improvement efforts.”
District leaders were frustrated with frequent changes to the performance metrics, for example, decreasing the weight of how much students grow academically in a year compared to what they score on the standardized test made it harder for them to get a high rating.
Another stakeholder described the performance ratings as “ranking and shaming on a scale from wealth to poverty.”
Untangling how much of a factor poverty plays in student outcomes isn’t something the audit answered.
“Are we measuring instruction or are we measuring poverty?” asked audit committee member Representative Dafna Michaelson Jenet.
The audit said differences in academic outcomes are not conclusive evidence that the accountability system is biased or unfair but acknowledged that poverty can play a role in student achievement.
It said differences in outcomes for these groups “could indicate the presence of unintended barriers or obstacles affecting their performance” or it could be due to “the quality of educational services.”
It also said that each student’s assessment score — administered two months before the end of school — is a point-in-time observation; “it is only an estimate of the student’s actual knowledge and skills.”
“What we're telling the parents is the student data is inaccurate but trust it anyways,” said Superintendent Don Haddad of the St. Vrain Valley School District, a leading critic of the accountability system.
“Pay attention when they say the system is doing exactly what it's designed to do. When you test the kids two months before the end of the year and you report it out as end of the year results, you have designed a test to ensure kids will not succeed.”
Districts react
Superintendents critical of the accountability system said the report is filled with examples showing there’s huge room for improvement.
Haddad said auditors concluded that the system provides a “reasonable” way to measure school performance.
“‘Reasonable’ to me is a very strategic word,” he said. “ I was looking for ‘effective,’ ‘reliable,’ ‘sound,’ not ‘reasonable.’”
He and others contend the design of the system is severely flawed and the report’s data supports that, particularly the fact that districts are consistently marked down for the test results for students living in poverty and students with disabilities even though across districts, including in high performance districts, those groups achieve at significantly lower levels compared to other students.
A red flag for him is the audit stated that smaller districts, because of their size, have less accurate results. That has sometimes landed them on the state’s watch list. About 153 of Colorado’s 178 districts are smaller districts.
“That should send off alarm bells,” he said. “That doesn't build any sense of confidence that what we're seeing is accurate. The majority of school districts are yielding inaccurate information.”
Oliver Grenham, chief education officer at Westminster Public Schools, found the audit’s findings “obvious.”
“They just looked at it as, ‘here's the results that are produced,’ but not asking why is this particular model producing these results? Is there something inherent in the model itself? And I believe that was the question that the legislature was really trying to get at.”
Though the audit concluded the system is improving outcomes, Grenham said the number of students reading and doing math on grade level has flatlined since 2010.
"So if your accountability system is improving the overall outcomes then you would expect the overall scores to be higher over time and they're not,” he said. “What the system is really good at is identifying who's on the lower end and not really at identifying why those people or those schools are on the lower end … So there's a bit of a denial around why gaps are continuing the way they are.”
Is state intervention helping low-performing students?
The audit found that low performing schools that had state-supported interventions generally experienced more gains or fewer losses in academic achievement, academic growth, and graduation rates than non-participating schools.
A total of 720 schools received the two lowest performance ratings at some point from 2010 to 2018. Only 18 percent remained at the lowest level for more than one or two years.
“Schools and districts receiving the lowest ratings do show improvement over time,” said Human Resources Research Organization’s Monica Gribben.
The audit found that schools that offer a higher number of advanced placement course offerings or a higher percentage of career and technical education graduates tended to have better student academic achievement, academic growth, and postsecondary and workforce readiness outcomes.
It also said schools serving higher proportions of students receiving free or reduced-priced lunch tended to have fewer advanced placement opportunities.
To many superintendents, this point in the audit underscores their point that funding matters. Colorado’s system has huge funding inequities between districts that can pass school tax measures and those that can’t.
“Those courses require funding,” said St. Vrain’s Haddad. “And so the inequity of funding is affecting the scores because they can't afford AP courses and technology and all the things that help the scores. And therein lies the bias. They're telling you that these things matter. They're telling you that socioeconomics matter. They're telling you that race matters. They're telling you that programs matter.”
The audit also found there was minimal difference between charter and traditional district-run schools with both having roughly the same number of schools in the top two performance categories.
Are performance measures helping educators and parents? Maybe and not really.
The audit included a voluntary survey for parents and educators. It showed that data needs to be more accessible, understandable and useful especially for parents. The majority of educators said they use state performance data to inform instruction, though nearly a third said it’s difficult for them to understand. State data also comes to them six months after students (in last year’s class) took the state tests. Seventy percent of parents said the system was difficult to understand.
Still, fewer than 1,500 educators responded to the survey, in a state with nearly 60,000 educators, including principals and superintendents. Just over 3,000 parents responded. In each case three-quarters of respondents were from Adams, Arapahoe, El Paso and Pueblo counties.
“I don’t know if I have a lot of confidence that as a state, teachers are using this data to inform student level instruction,” said Democratic state Sen. Rhonda Fields, noting just three educators in Denver responded to the survey. “I think there’s a message in the lack of response.”
Others took a different view.
“It does provide valuable information,” said Rep. Larson, a Republican from Jefferson County. “The overwhelming percentage of educators that did indicate some level of use for this data is valuable.”
Many districts have complained about the accountability system for a variety of reasons.
The complaints come back to that unanswered question, is the system biased against schools with high numbers of students living in poverty? Some district leaders said, yes, the measurements are unfair to districts serving more students of color and students who are low-income.
Take the Adams 14 school district, for instance. The state board of education’s order to, after more than a decade of low performance, reorganize the district in Commerce City – drew condemnation from a dozen superintendents in that area.
“We believe collectively in locally elected school boards of education and their Colorado constitutional right to the local control of the instruction of their students,” the superintendents wrote in a letter to the state board. A judge dismissed an Adams 14 lawsuit that tried to stop the order.
Others, like St. Vrain Valley's Superintendent Haddad, said the accreditation system has a number of faults, including incentivizing low credit requirements to graduate, which can boost a district’s graduation rate.
“We believe that it’s not a fair and transparent representation of the quality of schools,” he said. “It’s extremely limited and narrow and does not incentivize school districts to adequately prepare our children for the demands of this rapidly accelerating world we’re in.”
The St. Vrain district has implemented rigorous graduation standards and has made a massive investment in multiple career pathways, such as aeronautics, energy, culinary arts, law, medical and biosciences, and automotive training. But Haddad said when districts create a system that produces the kind of students’ employers want, they get penalized compared to a district that follows a narrower pathway for graduation standards.
Without a statewide graduation standard, districts can change the parameters for graduation. For example, during the pandemic, many districts' graduation rates actually went up, though thousands of students weren’t logging on to classes.
“Does a standardized test that is so limiting adequately and accurately reflect a child’s capacity for success in a global economy that is so technologically driven – because nothing on this assessment speaks to creativity, innovation, stick-to-it-iveness, teamwork?... so I think school districts are perhaps being labeled inaccurately,” Haddad said.
