Vance's CALL resources page | esl_home index
Return to Papyrus News Archive Main Page

Papyrus News
Online Reading, Testing, & Technology-Based Reform

I'd like to share a reading experience I had yesterday. It nicely ties together some threads related to critical literacy, online publication, and technology and school reform.

I was at a national conference last week and I heard mention of a report that supposedly outlines major gains in children's test scores in one state due to the use of computers. This state had supposedly risen from something like the 20th percentile to the 60th percentile in national test scores, and a group had supposedly proven that 1/3 of this gain was due to use of information technology. (I'm not mentioning the specific study because I'd prefer this critique to be more general rather than focused on a particular organization or researchers).

This was of course of great interest to me because everybody is looking for the definitive study that shows the impact of computers on learning. I found the URL for the study and eagerly read the paper.

The study, as it turns out, was terribly misleading. During the time period that the study focused on, which was over the course of one year, the gains in test scores were very slight (the study did its best to obscure how small those gains were, but they appeared to be somewhere between 1/2% and 2%). The study, using very questionable methodology and assumptions, claimed to show that 11% of this (to be generous) 2% gain was due to use of technology. So, even if the study's own questionable methodology could be believed, all it showed was that use of technology raised students' test scores in this state by about 0.2% (i.e., 1 point out of 500). Yet the study asserted much more than that, misleadlingly noting that "An 11% improvement in test scores would be welcomed by most students, parents, and educators," when no such 11% improvement in test scores occurred during the year, due to technology or for any other reason. (11% of the variance in determining a small gain was being confused with an 11% gain.) And other examples of misleading analysis and interpretation were evident throughout the report.

Well, why is this story worth telling?

First of all, in the days before the Internet, the chances of such an article getting a wide reading would have been much less. The report was published (and apparently commissioned) by a foundation, not through an academic outlet. The report had thus apparently gone through no independent peer review. Of course foundations published reports before the existence of the Internet, and people sometimes ordered or read those reports--or the reports were picked up in the press--so this is not a brand new phenomenon. But still, it is a good example of how the existence of the Internet makes low-cost publishing by a range of organizations and individuals that much easier. This is, for the most part, a good thing -- let 100 flowers bloom, as they say. But also weeds. So this is a good example of why the Web necessitates critical reading. This report had all the evidence of being authoritative. The authors had fancy degrees, it was published by a respected foundation, it had a detailed explanation of methodology--and it even had an approving postscript by a leading scholar (who now works for the foundation). But the report had not gone through the independent peer review that is required for academic journals, which almost certainly would have found its flaws. And these flaws might not be immediately obvious to someone without a background in quantitative research methods. These are the kind of flaws that we expect academic journals to sort out for us, but that aren't filtered out through this kind of publication process. So I'm all for breaking the monopoly of a few academic publishers, but I'm also for developing people's critical interpretive skills so they can recognize the difference between a peer-reviewed and an unreviewed report.

I'm not sure this story would be worth telling though if the only lesson was about online publishing, academic gatekeeping, and critical literacy. There are lots of other, probably more interesting, examples of the need for critical reading of online sources. For me though, the report also raises broader questions of testing and educational reform.

I believe that the national movement toward "high-stakes" multiple choice state testing is insidious and is having a profoundly negative affect on education. It is turning the focus of education away from anything meaningful, real, and critical, to a wasteful pursuit of tiny gains in test scores. This above study was about use of what is called an "independent learning system", in other words, an extensive system of computer-based exercises and drills (often referred to as "drill and kill"). It is probably one of the least interesting and least beneficial uses of technology in schooling, but it is a technology that matches well with the mindless pursuit of trivial gains in multiple-choice reading and mathematics tests. It is also a technology that is disproportionately used in the US with low-income students, whereas students in well-to-do neighborhoods have more opportunities to use technology for simulations, collaboration, creative production of knowledge, etc.) When technology is used in these more creative ways, it is much more difficult to try to demonstrate causation of a specific, measurable result on a standardized multiple choice test -- but the use of technology is beneficial nonetheless.

Let me close this by sharing a counter-example of use of technology. I had lunch the other day with an old friend and colleague, Susan Gaer, who has been a real leader in creative uses of technology in adult language and literacy courses. Her beginning ESL literacy students use technology for a whole range of practical and meaningful activities that integrating computer use, reading, writing, and project-based learning. They create their own business cards, they write and publish their own family stories, they do comparison shopping between online and community sources of groceries and pharmaceutical items, they make maps of their own communities. More advanced students research, edit, and publish an online journal. Her uses of technology are so varied--and particular to the needs of her own students--that it will be difficult to try to demonstrate a specific causal link to gains on a particular multiple choice test. But, in my opinion, these activities still offer a nice alternative to the use of independent learning systems. And her students are probably getting their first experiences in critical reading of online sources--something that they will apparently need even once they become highly literate!

Mark Warschauer

Use the navigator at the top of this page or your browser's BACK button to return to a previous page

For comments, suggestions, or further information on this site, contact Vance Stevens, webmaster. Regarding content of Papyrus-News, contact Mark Warschauer.

Last updated: April 22, 2001 in Hot Metal Pro 6.0