Cheating on the tests
Red Hat Rob reveals that the Tennessee Department of Education hasn't been honest about reporting the ACT scores for its students.
"...the Tennessee Department of Education takes advantage of the higher ACT scores by Non-Public School students by falsely reporting the average score forALL Tennessee students’ as if it were the average score for Public School Students."What got Rob started down this statistical labyrinth?
"I was hoping to discover the average ACT score for Tennessee homeschool students so that I could compare it to the average ACT score for Tennessee public school students. Sadly, the ACT Corp. doesn’t report the homeschool scores separately for individual states.The irony and outrage here is that the state Department of Education has the audacity to reject the high school diplomas issued by homeschools and church-related schools while at the same time using the higher ACT scores from those students to boost their own average."
Head over to Rob's blog to see what the real ACT scores are for Tennessee public school students.
7 comments:
Well, the state certainly shouldn't deceive the public in this way. I do want to add, however, that looking at ACT scores of only public school students doesn't give you the whole picture of public school performance either. Many of the best students, those headed for Ivy league or other out of state private institutions, take the SAT, not the ACT. So that will pull out some of what would be the best scores on the ACT. I think with the new state requirement, all students may be taking the ACT, but in recent years, either test can serve as the "exit exam" for MNPS students.
If only it did serve as the exit exam that would save a good bit of grief all around.
A bunch of stuff, here:
1. Obviously, it's not good for the state to use statistics that give potentially false impressions about this sort of thing. Clarity is always a virtue.
On the other hand, the speed with which non-public and anti-public school folks jump at the chance to call this cheating and deception says more about them than about the state, I think. It is, of course, possible that some fiendish character decided to milk a point or two out of the averages by putting in the composite, rather than the public-only statistic...or it's possible that someone slipped up and did so, without ever once twirling his mustache and laughing evilly.
(Case in point: I'm linking to a pro-homeschooling blog post on the Cat. IV issue, below. It also falsely refers to the overall average as the "government school" average.)
2. I dodged the ACT, myself, though I didn't get the sense that this was common among my peers. Just about everyone I talked to who had taken both the ACT and the SAT said that the former was either worse or easier to milk points out of on the same amount of knowledge. (Which I'd call worse.)
3. Just for the record, every time the Category IV diplomas issue comes up, I somehow end up following links to this blog post.
There are some interesting statistics in the article, actually, even though its tone towards public schools kids is a great example of why a lot of people get suspicious when homeschooling advocates get involved with public school issues and policy.
Specifically, according to the numbers there, homeschooled Tennesseans in '05, the only year where homeschoolers' statistics were clearly and separately recorded, scored an average of 20.7, just 0.2 higher than the average of all students. Moreover, those homeschoolers made up less than 1% of students taking the test.
Now, obviously, that's just a single data point. But the data that R.H.R links is strikingly static, especially in Tennessee, so I don't see any reason to assume that homeschoolers' scores have much improved. As it is, if R.H.R's calculations are correct, it seems likely that homeschoolers in '05 performed much more like public school students than, say, private school kids.
And it's worth noting that, with the less than 400 homeschoolers taking the test that year, outliers were able to play a much bigger role in the statistics. After all, the odds are good that there were a few incredibly bright, homeschooling students who probably could have blown the test out of the water if they'd been left alone, shackled to a well-stocked bookshelf. The odds are equally good that there were quite a few kids like that in the public schoolers' ranks, but those unusually high scores would have been balanced out by thousands and thousands closer to the average. The homeschooling outliers, on the contrary, had less than four hundred other scores to balance things out.
So, if we were to hypothetically set the outliers' scores aside for both, we wouldn't be amiss to suspect the overall average to remain relatively static and the homeschooler score to drop even closer to the public schoolers' average.
Given the relatively scanty numbers available, of course, none of this is certain. But the reverse argument, that TN homeschoolers do significantly better on the ACT than public school students, is equally unverified. (And, given the only recorded TN numbers, less likely.)I don't think anyone would deny that students going to established private schools in TN do a few points better, but that's a different story.
National homeschooling statistics are, as far as I can find, consistently one or two points higher than the national average. But national statistics have little bearing on the performance of specific states, and, just as TN's public schools are not redeemed by above average scores elsewhere, TN homeschoolers can't reasonably cite national averages to defend TN homeschoolers' scores.
And, nationally, you run into an even bigger risk of an outlier effect. The total number of public school students taking the test positively explodes, compared to the total number of homeschoolers, and there's no way to tell what data is actually representative of the possible benefits of homeschooling.
(Late-night Googling doesn't seem to turn up an official policy from ACT or the College Board, but I'm certain that I've read at least once that both caution against reading much into homeschool-specific data about their tests for this or similar reasons.)
So...N.S. Allen am I correct in assuming the following: if you take out the HS outliers they perform no better than the average non-academic magnet high school? If true, than all of the comprehensive high schools in Nashville (with one exception) have an average ACT score of less than 20. This does not meet the hope scholarship level and it does not meet college readiness levels...Is this how to intrepret the success of the average homeschooler in TN? I am confused. I really would like to know the average HS ACT score.
I went to the state report card site and pulled the ACT scores for NAshville's high schools. I found the following: the three year average ACT score for all MNPS high schools including MLK and HF is 23.7. If I exclude MLK and HF the three year average is 17.9. Hmmm...
Here is the breakdown by school --
Antioch 17.9
East Lit 18.7
Glencliff 17.
Hillsboro 20.2
Hillwood 19.1
Hunters Lane 17.4
Overton 19
Maplewood 16.3
McGavock 18.5
NSA 19.6
Pearl Cohn 16.3
Stratford 16.5
White Creek 16
HF 26.6
MLK 26.6
N.S. Allen, you need to understand that many homeschool students are Cat. IV school enrollees which the state classifies as private school students, not independent homeschoolers. I.e., many of your "private" students you speculate are outperforming homeschoolers are in fact homeschool students. There is no way for you to determine the mix of Cat IV "brick and mortar" students vs. Cat. IV homeschool students either, although you can determine which Cat IV schools do not have this arrangement at all. BTW, the largest enrollments for Cat. IV schools are the ones with the satellite option.
As for 'outliers', I would suggest that homeschool and private students do not have near the dropout rate that the public system does, thus when TN states that 93% of graduates took the ACT, you are only talking about 60% of the students even attempting the test (See Kay's earlier posts about the fishy dropout/graduation rates reported by the state). It is the public school system that is testing with "outliers". Some systems are testing less than half of their seniors. I don't think you will find that to be the case with private or homeschool students of any stripe.
Finally, assuming you are correct that there is "no way to tell" the benefits of homeschooling, you have merely proven Rob's point, which is: how then can the state so confidently state that their "diplomas aren't worth the paper they are written on"?
RE: din819go, I think I should have been more specific as to the "outliers." There are, after all, scores markedly different from the average that still give us important information.
For instance, you note that removing HFA's and MLK's scores from the high school averages causes it to drop by quite a few points. But that's because HFA and MLK are better schools than most other public schools in MNPS. Since we're trying to determine the benefits of home-schooling relative to public-schooling, it makes no sense to exclude the scores of the best public schools, just as we wouldn't want to exclude, say, homeschoolers whose parents are particularly good at making their house into an effective classroom.
My point was simply that, in the comparatively tiny home-schooling set, natural differences in ability, rather than formal education, could have had a big effect on the average. If, for instance, I have two kids who are geniuses, one in public schools and the other at home, both are likely going to do very well on the ACT. But, since these students have an advantage unrelated to the form of schooling they've received, their scores don't tell us much about the efficacy of those forms, unless we compare just the geniuses in both sets.
When we look at the averages, though, the huge number of public school students taking the tests will inevitably drown out those natural outliers. The comparatively tiny homeschooling set, though, could have its average score shifted well away from the actual, standard performance of Joe Homeschooler on the test.
(Obviously, accounting for these factors is nearly impossible with the statistics we have or even with statistics that the ACT could plausibly gather. But the mechanism by which these factors could become an issue are straight-forward enough and should be kept in mind.)
RE: Eric, obviously, the Category IV standard is decidedly muddled, and it makes it a bit hard to say, for Category IV purposes, who we should call private and who we should call homeschoolers.
I don't think it makes much of a difference regarding the numbers, though. The ACT listed separate scores for homeschoolers in '05, and I strongly suspect that inclusion of one's scores in that category were based on self-identification. So, at least for TN homeschoolers who consider themselves such, the statistics are as the ACT presents them.
It's fair enough to say that we can't directly apply these numbers to the broader "Category IV schools" group, as a whole. Did some of those kids not self-identify as homeschoolers, for instance? If that's the case, the statistics would be muddled with regards to the subset of Category IV students who don't identify as homeschoolers. And maybe they're part of the group of private school students who are getting the significantly higher scores that Rob deems necessary. We just don't know.
As to what this says about the Category IV issue as a whole - not much, I think. Other people can debate whether the stats we have mean that the state's positions on Cat IV diplomas is justified or not. I think, personally, that the ACT issue is almost entirely tangential to that question, and, at least here, was simply trying to make a point re: the effect of homeschoolers on total ACT averages for TN.
(I, for my part, think the DOE's position is just fine, but I don't think ACT scores are evidence for that position or its reverse.)
"the Category IV standard is decidedly muddled, and it makes it a bit hard to say, for Category IV purposes, who we should call private and who we should call homeschoolers."
Actually it isn't muddled at all. They are all private school students.
"Did some of those kids not self-identify as homeschoolers, for instance?"
Did some of the public students call themselves homeschoolers? I am not sure why you belabor this 2005 single data point that you have previously called "clearly and separately recorded" and now are calling doubt upon. Either the 2005 data is good, which shows a group of homeschool students outperformed public school students on one test, or the data is not in which case neither side can use it. However, it is the only data the state can look to at this time, completely ignore and then declare that non-public student diplomas "aren't worth the paper they are written on". I am not sure if this is the DOE position you say you agree with, but if it is, I would like to continue the discussion regarding what your evidence in holding this position is - since the ACT scores are off the table either way for you.
"I have two kids who are geniuses, one in public schools and the other at home, both are likely going to do very well on the ACT."
This is an often-used misconception - that homeschool students do well on standardized tests, spelling bees, etc. therefore they must all be geniuses. This is a common fallacious tact taken by institutional education, because it simultaneously explains the better performance of the students while discrediting the teaching ability of the parent. It must be the raw intellectual ability overcoming the sub-standard environment and teaching ability. The facts are that we do not know what percentage of homeschool students are taking the ACT, but we do know that only roughly the top academic third of public students are taking it - that would include all your theoretical public school geniuses and significantly narrows the outlier gap.
Frankly, I do not believe there are that many geniuses. I believe many (if not all) parents and students can produce "genius-like" results with some effort. I.e., when my 9-year old scores "post-high school" for reading grade equivalent on a nationally normed achievement test, I don't think they are a genius. I think of what poor readers we have that the average reading score of a post-high school student would be the same as my 3rd grader.
Of course, TN hides in its TCAP bubble so they cannot be compared to anyone else (say vs. a Stanford 10 nationally normed exam) so I would have no idea how a private student in TN compares to a public student or how that public student compares to the rest of the country. It is only when TN public students take tests related to NAEP that the bubble gets popped.
See pages 6-7: tennessee.gov/sbe/TDP%201-23-08.pdf
These proficiency deceivers are the same people who with no data are claiming that non-public education is worthless.
Post a Comment