The Daily News analysis comes as pressure mounts to boost the scores used to rank schools, evaluate teachers and mark the academic progress of 980,000 Ohio students each year. Officials acknowledge that when scores climb — even by improbable leaps — the emphasis is on identifying what teaching strategies were successful, not on investigating whether cheating might have played a role.
The analysis found that more than 500 districts and charter schools in Ohio had at least one school with an improbably large score change from 2005-2011. In 42 districts and charters, the probability of so many big swings happening by chance in any one year was less than one in 1,000, or less than one-tenth of 1 percent.
The Daily News analysis was done in partnership with The Atlanta Journal-Constitution. The two newspapers collaborated on what is believed to be the first-ever examination of standardized test scores in all 50 states.
The newspapers used a statistical model on year-to-year changes in reading and math test scores at the school level for third- through eighth-grade students during seven years, beginning in 2005.
The analysis flagged as suspicious any score change that had less than a 5 percent probability of occurring by chance based on all the other scores on that test in that state.
The study then calculated the probability that any district would have the number of flagged or improbable scores it had in any one year. In some cases, those probabilities were approaching zero.
The Ohio Department of Education and Ohio Education Association, a union that represents 125,000 teachers, both questioned the validity of the Dayton Daily News/Atlanta Journal-Constitution methodology and results.
“We do have problems with your methodology and we don’t necessarily subscribe to the viewpoint that your methodology has exposed any cheating in Ohio,” said Jim Herrholtz, ODE associate superintendent for learning.
OEA Spokesman Mike Mahoney said: “We feel that there is a grave chance that this data and this study are totally flawed. And you really ought to hesitate before going forward and harming the reputation of the nation’s public schools, teachers and these school districts, who are working very hard for students’ success, before you publish this kind of thing.”
The Daily News is holding off on naming individual districts or schools until more analysis can be done on scores in the flagged classes.
The teachers union criticized the analysis for not doing enough to account for individual student changes within a grade from year to year. Individuals can move in and out of schools during the school year, and that mobility is higher in poorer urban schools than in a wealthier suburban district. In Cleveland, for example, 31 percent of students attended two or more schools during the 2010-11 school year.
Because of the mobility issue, Western Michigan University education professor Gary Miron said the study is more likely to flag poorer schools.
“Although there may be some schools in the mix that are involved in systematic cheating, most schools will be targeted because they have lots of students coming and going from year to year,” Miron said.
However, more than 2,600 improbable changes — large spikes or drops — were found in Ohio schools between 2005-2011. Because of the sheer number, it seems unlikely all of them can be explained away by the quality of instruction, demographics, or changes in mobility and class size. Similar swings were evident in Atlanta and other cities where cheating was uncovered.
Jaxk Reeves, a statistics professor and director of the Statistical Consulting Center at the University of Georgia who helped with the newspapers’ analysis, called the methodology a “reasonable” way to identify curious changes in test scores in a grade, subject and school from one year to the next.
Unless the boundaries of a district changes to suddenly bring in wealthier or poorer students, the average students coming and going in a district are going to look pretty much the same, he said.
“It’s not going to make that much difference,” Reeves said of tracking individual students. “It would be a little bit nice to know. But the way you’re doing the analysis right now, it won’t affect much.”
Reeves emphasized that an initial flag on a test score does not prove guilt. Having a lot of flags, however, is reason to explore further to see what is causing such wide swings, he said.
“This technique is designed to give you some general picture,” said James Wollack, director of testing and evaluation services in the Department of Educational Psychology at the University of Wisconsin-Madison, who also consulted on the project.
“You would not use the results of this kind of analysis to say definitively that any of the schools that show up on the list would be guilty of misconduct. It would trigger an investigation.
“So long as this is understood, then I really don’t have a problem with this methodology.”
When a pattern of spikes and drops showed up in the Atlanta Public Schools, a probe by the state of Georgia uncovered an extensive scheme of cheating that involved 178 educators at every level in the school district.
“We are not Atlanta,” said Herrholtz, who has served as superintendent in three Ohio districts.
How schools cheat on test scores
Though not to the level of an Atlanta, some Ohio schools have been caught cheating.
In 2006, the Daily News exposed that huge gains in test scores at City Day Community School in Dayton came when teachers drilled students on practice tests with identical or substantially the same questions as appeared on the actual state exam days later. City Day scores plummeted the year after the cheating came to light.
The Ohio Department of Education said it guards against cheating by requiring its testing contractor, American Institutes for Research, to analyze how often test-takers rub out one answer and choose another on the fill-in-the-bubble answer sheets. Unusually high change rates may indicate cheating, particularly when it shows switching to the right answer.
Experts say this analysis doesn’t catch other cheating methods. A Daily News examination of disciplinary actions show a wide range of cheating tactics, including leaving a test at a student’s home, drawing visual aids on a blackboard, leaving test booklets unattended and returning incomplete tests to students so they could finish up.
ODE prides itself on extensive training for educators on test protocols and on a culture of districts blowing the whistle on themselves when problems arise.
“Ohio has been very vigilant and we have a strong track record of the self-reporting culture that is out there in the field. And we have taken swift action if we have any incidents of cheating or of test security breaches,” Herrholtz said. He noted that Ohio teachers know their licenses and jobs are potentially on the line “if they attempt, in any way, shape or form to cheat.”
The state’s safeguards have yielded the following: 60 licensed educators disciplined since 2007. It’s not clear how many allegations of cheating were reported during that time, but some came from anonymous phone calls, some were flagged cases by the test vendor and some were self-reported by the schools, according to ODE officials.
In the last two years, 21 of the 33 allegations involved self-reporting, ODE said.
Relatively few scores are flagged by the state’s test vendor, in part because the state sets a high statistical bar before changes in test scores get attention.
The state requires AIR to flag scores that fall four or more standard deviations from the mean. That standard would draw attention to fewer than one in 10,000 scores in a random distribution of scores. Other school testing analyses conducted by news organizations around the country have used two or three standard deviations to flag scores. Three standard deviations would draw attention to about one in every 1,000 scores.
Miron was surprised that the state spends so little on erasure analysis and sets the threshold at four standard deviations, which he called too high. He said states and districts willing to pay more can get testing contractors to do more rigorous analysis.
“The test companies can do all these things. ... They can write the programs that will systematically create red flags that will warn the education agency, whether it’s a state agency or a district that’s contracting with them. And they will know where there is warranted attention for further investigation,” Miron said.
He added, “There is so much pressure to cheat right now because of high-stakes testing, because the consequences for schools, and also increasing consequences for teachers. So there is going to be more cheating and we have to have these safeguards in place.”
Herrholtz said Ohio’s flagging criteria is based on recommendations from nationally recognized experts.
Ohio School Boards Association Executive Director Damon Asbury, who served as superintendent of Worthington and Columbus schools, said 99.5 percent of teachers act with the highest professional standards; but districts may want to look at the test score data more closely.
“If the facts are there and the patterns are there, you have to — what’s the old Ronald Reagan line? — trust but verify,” Asbury said.
The state is re-working the standardized testing system in conjunction with 27 other states and will switch from paper and pencil tests to online assessments by the 2014-15 school year, thus eliminating any ability to do an erasure analysis on the results. Ohio and other states share the same concerns that “as we ratchet up our common core standards and we do raise the bar for students for passage, we certainly raise the risk of there could be some motivation to cheat, if you want to call it that way,” Herrholtz said. “So we are going to have to have good safeguards in place to ensure that that doesn’t occur.”
The presidents of the Ohio’s two largest teachers unions emphatically made three points: cheating is wrong, the vast majority of teachers are striving to give students the most well-rounded education possible, and the state places too much emphasis on the state achievement test.
“My opinion is that we are putting too much emphasis on one test and that’s why you are seeing problems happen,” said Melissa Cropper, the newly elected president of the Ohio Federation of Teachers, which represents 20,000 members. She said Ohio needs to augment the standardized tests with other measures of student performance. “That to me is the problem, not whether people are cheating on the test or anything else,” she said.
Allegations are shrouded in secrecy
Allegations of cheating are kept private at the state level. The Ohio Board of Education in December 2007 mandated that all information relating to investigations into violations of test security be confidential forever, even if findings are made.
Susan Tave Zelman, who served as state superintendent at the time, said it was written that way to protect educators against “vindictive and bogus” allegations that could unjustly smear their professional reputations.
Even when educators are disciplined, only minimal details are made public. The Daily News review of disciplinary cases showed most resulted in letters of admonishment or suspensions. Just one resulted in a license revocation. Only scant details are ever released.
Employee personnel files don’t always document what happened either. In October 2009, Christine Hatton, then principal of Mills Lawn Elementary School in Yellow Springs, abruptly resigned for “personal reasons” after eight years in the district. Five months later, the State Board of Education suspended Hatton’s license for 12 months, beginning Oct. 31, 2009, for “inappropriately assisting students while administering the Ohio 4th Grade Achievement Test at Yellow Springs Village School District.”
Yellow Springs Superintendent Mario Basora said Hatton’s personnel file does not detail the transgression.
Tony Armocida, who was superintendent when Hatton left, said, “She resigned and she was disciplined by the state. That’s as much as I’m going to say.”
Hatton did not return multiple telephone messages.
Parents with students at Mills Lawn Elementary were not told why Hatton departed. “We were told it was a private matter,” said Violette-Anne Curley. “We were not happy about no one telling us what happened because it involved our kids at the school. ... She was paid with our tax dollars, too. It’d be nice to know what the people we hire and pay do.”
State Sen. Peggy Lehner, R-Kettering, who chairs the Senate Education Committee, said, “I would certainly think parents should have a right to know if it involves their school.”
Cleveland Metropolitan Schools Chief Executive Eric Gordon isn’t happy with his district’s scores, which are among the worst in the state, but that has little to do with a concern over cheating. “In all candor, if I thought we had a widespread cheating problem in the district, I would expect our achievement to look quite a bit better than it does,” Gordon said.
Wollack said the pressure to improve test scores creates a disincentive for districts to look for cheating. “I don’t think there’s any question about that,” said Wollack, who specializes in test security and detection of cheating. “I think it’s a huge conflict of interest.”
At least one of those caught, former teacher Ophelia Fry, blamed her actions on the pressure to raise scores.
Mansfield City Schools fired Fry, a sixth-grade teacher at Malabar Middle School, in 2006 for violating testing procedures. In 2009, the state school board put a letter of admonishment in Fry’s licensing file for encouraging students to erase, cross out and rewrite answers on state tests. Fry fought the dismissal and filed an age discrimination lawsuit in federal court, citing examples of younger teachers who she said did virtually the same thing without penalty. She lost on both fronts and retired in 2007.
In an interview, Fry said she and other teachers were under pressure from the building principal to boost scores and leverage the district out of academic emergency. She said her boss told them to tell kids to fill in an answer for every question and send them back to their desks to try again if they left any blank.
“So if I saw a blank answer, I would say to the kid, ‘You can’t leave this blank. You have got to write something in there.’ I never, never gave them an answer. That is something I would not do,” Fry said. “I felt as though my job would be in jeopardy if I didn’t adhere to what she was asking,” Fry said of her boss.
Pressure fueled Atlanta scandal
It was that type of pressure, along with financial incentives, that helped fuel the nation’s worst cheating scandal: Atlanta.
In 2008 and 2009, The Atlanta Journal-Constitution published analyses that found suspiciously high gains on state assessment tests in Atlanta Public Schools. In July 2011, Georgia special investigators issued a scathing report detailing potentially illegal behavior in a cheating scheme.
In education circles, “Atlanta” is shorthand for unethical behavior and systemic failure.
David Quolke, president of the Cleveland Teachers Union, said he is unsure whether Ohio needs to do more to monitor big swings in test scores. But, he said, “Clearly, if there is a problem, it has got to be addressed. No one wants an Atlanta.”
About the Author
