Uses and Misuses of Processing Tests
By: Louise Spear-Swerling (1998)
If you have ever been involved in an educational evaluation for learning disabilities -- whether as a parent, teacher, or in some other role -- you have probably heard about "processing tests" and "processing disorders." Many aspects of these evaluations can be problematic, but perhaps none more so than those entailing the use (and potential misuse) of processing measures. Yet the right measures, carefully interpreted, can be enormously helpful in education. In this article, I will discuss the benefits of processing measures, as well as their limitations. Although I will focus primarily on the area of reading, the possible uses and misuses of processing measures are similar across other domains of learning disabilities, such as mathematics and written expression.
How processing measures are used in education
The Individuals with Disabilities Education Act, or IDEA, defines learning disabilities as involving "a disorder in one or more of the basic psychological processes." Many state regulations also include the concept of a processing disorder in their definitions of learning disabilities. In education, determining the presence of a processing disorder generally involves using tests of memory, language, auditory processing, or visual processing---abilities believed to underlie academic learning---rather than tests with actual academic content. Although educators often employ the latter types of tests to document an IQ-achievement discrepancy or to evaluate progress, they usually do not view these tests as measures of processing.
However, cognitive psychologists have conceptualized processing measures more broadly than have many educators. For example, cognitive psychologists interested in reading disability (or dyslexia) have emphasized that a core deficit for dyslexics appears to involve the use of phonological processes in reading (Rack, Snowling, & Olson, 1992). A measure widely used by scientific investigators to assess these abilities is the Word Attack subtest of the Woodcock Reading Mastery Tests, which requires the individual to read out-of-context pseudowords (nonsense words) such as taff and grum. In contrast, educators would more commonly use this subtest to document low reading achievement than as evidence of a "processing disorder."
Here I will use the term "processing" broadly to include tests with, as well as without, obvious academic content.
How processing tests can be helpful
Processing measures can be quite helpful in early identification of reading difficulties and in planning reading instruction. Trying to teach a child to read without information about the underlying cognitive processes important in reading is like trying to figure out how your car works without being able to look under the hood. An essential point, however, is that the measures used have to be the right ones. Just as looking in the trunk probably won't tell you much about whether your car engine is working properly---and if not, what the problem is---using the wrong processing measures won't provide much insight into children's (or adults') reading difficulties. For poor readers, the most relevant processing tests tend to be measures directly related to reading (e.g., letter identification or out-of-context word decoding) or measures of linguistic processes (e.g., phonological awareness). Also, the tests used in any educational evaluation must be technically adequate; for instance, they must have adequate norms, reliability, and validity. (Potential sources of information about the technical adequacy of specific tests can be found at the end of this article.)
Of course, the processing measures most relevant to mathematics or written expression are not identical to those most useful in reading. Thus, in choosing processing measures, an examiner should consider the academic domain(s) in which a child experiences difficulty. Unfortunately, in education, processing measures are not necessarily selected on this basis. Rather, it is quite common for all children at a particular age level to be given the same, usually fairly extensive, battery of processing measures; a low score on any measure then may be assumed to reflect a "processing disorder." One problem with this scattershot approach, in addition to the fact that it may not yield educationally useful information, is that it capitalizes on chance. In other words, if enough tests are given to a person, even to someone with no learning problems at all, he or she probably will eventually get a low score on at least one test. This possibility is especially likely in educational evaluations for learning disabilities because many of the tests used in these evaluations lack technical adequacy (Moats & Lyon, 1993). In this situation, an individual's low score on a processing measure may be completely meaningless.
I. Early identification of reading difficulties
Much of the research on early identification of reading difficulties has focused on using measures of various cognitive abilities to predict future reading achievement in preschool or kindergarten children who are nonreaders. This work suggests that the following three abilities strongly predict beginning reading achievement: letter identification, phonological awareness (or sensitivity to the sounds in spoken words), and the capacity to rapidly name arrays of letters, digits, colors, or pictured objects (Wagner & Torgesen, 1987; Wolf, 1991). In addition, in a very thorough review of the literature on early identification, Scarborough (1998) found that expressive vocabulary, sometimes called confrontation naming, is highly predictive of early reading achievement. Expressive vocabulary is usually assessed by asking children to name individual drawings or pictures of objects. Finally, general language ability also predicts reading achievement, and children with broad language difficulties or delays clearly are at risk for reading problems.
Although these reading-related cognitive measures are quite useful in predicting reading achievement for large groups of youngsters, they fall far short of perfection in individual screening. For instance, on average in the studies reviewed by Scarborough (1998), even screening batteries involving combinations of tests failed to identify over 20% of kindergartners who would later go on to experience reading difficulties; and they erroneously identified an even larger percentage of kindergartners as "at risk" who would turn out not to have reading problems at all. Nevertheless, if their shortcomings are recognized, these measures can still be useful in educational testing. And as Scarborough points out, routine screening using these tests might well be worthwhile for young children already known to be at risk for reading difficulties, such as those with oral language impairments or a family history of reading problems.
At least one other type of measure can be helpful in early identification of somewhat older children who have experienced formal reading instruction. This measure involves out-of-context word decoding, as on the Word Attack subtest of the Woodcock Reading Mastery Tests, mentioned earlier. The use of pseudowords, such as glift, is important to provide a measure of pure decoding as opposed to sight-word recognition. It appears that children with significant weaknesses in out-of-context word decoding are at risk for future reading problems, even when they have adequate sight-word knowledge and can read better in context than in isolation (e.g., Byrne, Freebody, & Gates, 1992). The problem for these youngsters is that, as they advance in school and the texts that they have to read become harder, it becomes increasingly difficult to compensate for poor word decoding by using sight-word or contextual abilities.
II. Instructional planning in reading
One way that processing measures can be useful in instructional planning is in pinpointing weak areas in need of direct instruction. For instance, there is abundant evidence that training phonological awareness, especially in conjunction with a program of formal reading instruction, can improve reading achievement in many at-risk youngsters (e.g., Hatcher, Hulme, & Ellis, 1994). Similarly, cognitive strategy instruction, especially when integrated with content instruction, may benefit reading comprehension in older children (Ellis, 1993). And intensive, systematic instruction in word decoding improves decoding accuracy both in young at-risk youngsters and in older poor readers (Torgesen, 1998). (However, as discussed further below, it should not be assumed that training of all weak areas is warranted.)
A second way that processing measures may be useful in educational planning is in giving information about an individual's overall cognitive profile in reading, which in turn may yield implications for instruction. For example, a youngster who obtains low scores on reading-comprehension tests, but who has an above-average verbal IQ and good listening comprehension, is not likely to have a true comprehension problem. Rather, it is more likely that this kind of youngster has problems revolving around word decoding that is inaccurate, nonautomatic, or both. In this case, in order to improve reading comprehension, the educational program must build accuracy and fluency of word decoding.
Table 1 summarizes some of the cognitive processes discussed so far, along with examples of tasks and individually-administered standardized tests that often are used in evaluating each type of ability.
|Cognitive Process||Sample Tasks||Examples of Standardized Tests|
|letter identification||name single letters and/or give their sounds||Woodcock Reading Mastery Tests (WRMT) Letter Identification|
|phonological awareness||segment a spoken word into individual phonemes (phoneme segmentation); identify spoken words that begin with the same initial sound (alliteration); blend orally presented phonemes into a whole word (phoneme blending)||Lindamood Auditory Conceptualization Test (LAC); Test of Phonological Awareness (TOPA)|
|rapid naming||name as quickly as possible an array of letters, single digits, colors, or pictured objects||Rapid Automatized Naming (RAN) (Denckla & Rudel, 1976)|
|expressive vocabulary||name individually presented line drawings or pictures||Boston Naming Test (BNT)|
|receptive vocabulary||given a set of pictures, point to the correct one when the examiner names it||Peabody Picture Vocabulary Test (PPVT)|
|word decoding||read aloud out-of-context pseudowords (e.g., taff)||Woodcock or Woodcock-Johnson Word Attack|
|speed and fluency of reading||read words or paragraphs aloud under timed conditions; observations of ease and expression of reading are also made||Gray Oral Reading Test (GORT)|
|use of comprehension strategies||answer questions about use of strategies during reading (e.g., when comprehension fails, does the individual make use of strategies such as rereading?); observations of strategy use are also made||usually assessed informally|
|listening comprehension||listen to words, sentences, or paragraphs read aloud by the examiner, then point to a corresponding picture or answer verbally presented questions about the material heard||Wechsler Individual Achievement Test (WIAT) Listening Comprehension|
|reading comprehension||read a sentence or paragraph with a blank in it and fill in a word that makes sense (cloze format); answer questions about material the individual has read aloud or silently||Woodcock or Woodcock-Johnson Passage Comprehension (uses cloze format); Gray Oral Reading Test (GORT) and WIAT Reading Comprehension (both use question-answering format)|
The limitations of processing measures
Although the right processing measures can be very valuable in early identification of reading problems and in educational planning, these measures also have some important limitations, such as the following:
I. Processing measures cannot be used to determine the ultimate causation of an individual's learning problems
In educational evaluations for learning disabilities, poor performance on processing measures frequently is assumed to result entirely from an intrinsic deficit or disability. However, processing tests are at the level of psychological description; they cannot be used to determine whether an individual's learning problems are primarily the result of "nature" or "nurture." For example, although phonological-processing abilities such as phonological awareness appear to be strongly genetically influenced, they also are shaped by environmental factors (Mann, 1994; Olson, Rack, Conners, DeFries, & Fulker, 1991), such as home literacy experiences and the nature of classroom reading instruction. Behavioral geneticists interested in reading disability have emphasized that genetic influences never develop in isolation, but always in interaction with the environment, both the surrounding biological environment (i.e., the individual child's unique biological make-up) and day-to-day experiences. Moreover, particularly in older children and adults, some processing weaknesses actually may be caused by poor reading itself. For example, a lack of exposure to text---because of failure to engage in independent reading---may lead to problems with orthographic-processing tasks, such as those requiring spelling knowledge or reading of phonetically irregular words (Stanovich, Siegel, Gottardo, Chiappe, & Sidhu, 1997).
Current federal and state educational guidelines charge learning-disabilities specialists with sorting genuine cases of learning disabilities from other types of achievement problems. Unfortunately, however, at present it is impossible for special educators to meet this charge, at least for most cases of poor reading. Although there is a large research literature on the biological underpinnings of reading disability, most measures that have been employed in this research are not feasible to use in educational screening, nor could they yet be used to separate genuine cases of reading disability from other types of reading failure. And processing tests cannot be used to identify genuine cases of reading disability, because no unique processing profile has been associated with true reading disability, and because poor performance on processing tests is not proof of an intrinsic disorder in learning.
II. Interpretation of processing measures can be difficult.
Tests that appear to measure primarily one set of processes may actually tap very different processes. For example, most of the visual-processing measures used in education involve paper-and-pencil tests or relatively long exposures of test items. These measures often tap linguistic, memory, or motor abilities instead of---or in addition to---visual processing (Vellutino, 1979). These problems make it impossible to know whether children who obtain low scores on educational tests of visual processing actually have a visual-processing weakness, or some other type of weakness. For these reasons, scientists interested in the role of visual processes in reading have developed purer measures of visual processing, often involving tachistoscopic presentation of test items. However, these kinds of measures are rarely, if ever, used in education.
Likewise, Scarborough (1998) suggests that tests of expressive vocabulary may predict beginning reading achievement because they tap phonological processes important in early reading---not necessarily because expressive vocabulary itself is essential to early reading.
Even processing measures with academic content, whose interpretation may appear to be straightforward, do not always measure exactly what they are believed to measure. For example, Nation and Snowling (1997) studied two different tests of reading comprehension, a measure that used a sentence completion or cloze format (in which the child has to read a sentence and fill in a blank with a word that makes sense, e.g., "The fish were swimming in the ______") and a different measure that required children to read stories aloud and answer questions about them. They found that the former measure, using the sentence-completion format, related more strongly to word recognition than to broad language comprehension; only the measure involving reading stories and answering questions about them related strongly to language comprehension.
In other words, although both types of tests were labeled as measures of "reading comprehension," they were actually tapping different underlying processes---because of differences in the tasks they used. In fact, Nation and Snowling found some children with significant language-comprehension weaknesses who scored within the average range on the sentence-completion test. Conversely, they point out that a dyslexic youngster might get a low score on the sentence-completion test and be erroneously assumed to have a reading-comprehension problem, when his or her problems really center upon poor word decoding.
III. Considerable caution is required in using information from processing measures in instructional planning.
One of the reasons that there has been so much enthusiasm in the scientific community about phonological awareness is that, not only has phonological awareness been shown to predict reading achievement, but training it has actually proved beneficial to reading. As mentioned earlier, strategic abilities in reading also appear to be trainable. Unfortunately, however, we still know little about how, or even whether, to train many reading-related cognitive processes.
For example, although rapid naming tasks predict early reading achievement, currently there is little evidence that directly training those tasks improves reading. Most scientific investigators of reading are quite guarded about recommending training of cognitive processes to educators. In order to make training worthwhile, educators need to know that the processes in question have a causal relationship to reading and are trainable, as well as that training them improves reading achievement. This knowledge is not yet available for many of the cognitive processes that have been studied in reading.
Finally, information from processing measures sometimes is used to match a method of reading instruction to a child's underlying "modality preference" or "reading style." For instance, a youngster with visual-processing strengths and auditory-processing weaknesses might be prescribed a sight-word or holistic reading method, whereas a youngster with the opposite profile (i.e., strong auditory, weak visual) might be prescribed phonics. Research literature reviews have repeatedly concluded that matching reading method to modality preference is not effective (Arter & Jenkins, 1977; Kavale, Hirshoren, & Forness, 1998). Furthermore, it should be noted that many poor readers appear to have auditory-processing weaknesses on educational tests---because these tests often use linguistic stimuli such as words or numbers, and poor readers frequently have phonological-processing problems that influence their performance on linguistic tasks. (In other words, what may be interpreted as a global auditory-processing deficit may really be a phonological-processing deficit.) If these youngsters are prescribed a pure sight-word or holistic method instead of explicit teaching of word decoding (i.e., phonics), their problems may be compounded.
IV. There is much more to achievement than what is captured by conventional processing measures.
Processing measures do not tap, or only indirectly measure, many individual differences that ultimately may be quite important to achievement. For instance, educational researchers (e.g., Baumann & Duffy, 1997; Farkas, 1996) have found that noncognitive variables such as persistence, work habits, interests, and motivation, play an important role in reading achievement for children from a range of backgrounds and ability levels. Moreover, noncognitive variables may influence the development of cognitive abilities over time, because individuals who are motivated to persist at certain tasks---such as reading difficult books---are more likely to improve the cognitive abilities needed for those tasks. Similarly, cognitive abilities such as creativity and practical intelligence (e.g., Sternberg, 1996) usually are not tapped by traditional educational measures, but may strongly influence an individual's success in life. Thus, even if a youngster has a significant and enduring deficit in some processing ability---and there is evidence to indicate that many individuals with reading disability may have this kind of long-term weakness in phonological processing---this deficit does not necessarily preclude going to college, pursuing a profession, or having a satisfying, successful life.
Of course, teachers and parents should still be concerned about signs of reading difficulties and should do all they can to address these problems. However, it also is helpful to keep a sense of perspective about reading difficulties---and to try to communicate this perspective to youngsters who are struggling in school.
How processing measures should be used in educational evaluations
To sum up, here are some characteristics to look for in educational evaluations involving processing measures:
The measures should tap important component processes in the individual's domain(s) of difficulty. In reading, the cognitive processes that are most essential to consider may vary somewhat with age and level of reading achievement. For young and beginning readers, important component processes include phonological awareness, letter identification, listening comprehension, and accuracy of decoding single words. For older and more advanced readers, it may be especially important to consider speed and fluency of reading, the use of comprehension strategies, and detailed analyses of reading comprehension (e.g., literal vs. inferential comprehension, or comprehension of narratives vs. expository text).
Processing measures should be technically adequate. Information on the technical adequacy of many commercial tests can be found in sources such as Buros' mental measurements yearbooks, Sweetland and Keyser (1993), basic textbooks on special-education assessment (e.g., Salvia & Ysseldyke, 1995), and in the test manual itself.
Processing measures should not be the basis for assumptions about the ultimate causation of an individual's difficulties. Instead, they should be viewed as a source of educationally useful information about which children may be at particular risk for reading problems and about an individual's current pattern of strengths and weaknesses in various reading-related cognitive processes.
Information from processing tests should make sense. This information should be consistent with, but should also complement and clarify, observations of the individual's everyday performance in reading. If test results cannot be reconciled with impartial, well-informed observations of day-to-day performance, the accuracy of test results should be questioned.
This information also should lead to specific, educationally useful recommendations that are tailored to an individual's needs. Very broad recommendations, such as "needs structure" or "needs to learn organizational skills" apply to almost all low achievers (and to some high achievers as well). More useful are recommendations such as "needs to learn strategies for decoding multisyllable words," or "needs to build speed and fluency of reading in context," along with specific techniques or programs for meeting these goals.
Certain pitfalls in using processing measures in instructional programming should be avoided. These pitfalls include training of cognitive processes in total isolation from academic instruction, attempted training of processes that have not been shown to be trainable, and matching reading method to modality preference.
About the author: Louise Spear-Swerling is a professor of special education at Southern Connecticut State University. She has taught both special education and regular classes in public schools and now teaches graduate and undergraduate courses in learning disabilities. She is also the co-author of Off Track: When Poor Readers Become 'Learning Disabled'(Westview Press, ©1996, ISBN 0-8133-8756-6).
Click the "References" link above to hide these references.
Arter, J., & Jenkins, J. (1977). Differential diagnosis: Prescriptive teaching---A critical appraisal. Review of Educational Research, 49, 517-555.
Baumann, J. F., & Duffy, A. M. (1997). Engaged reading for pleasure and learning: A report from the National Reading Research Center. Athens, GA: National Reading Research Center.
Byrne, B., Freebody, P., & Gates, A. (1992). Longitudinal data on the relations of word-reading strategies to comprehension. Reading Research Quarterly, 27, 140-151.
Denckla, M. B., & Rudel, R. (1976). Rapid automatized naming (RAN): Dyslexia differentiated from other learning disabilities. Neuropsychologia, 14, 471-479.
Ellis, E. S. (1993). Integrative strategy instruction: A potential model for teaching content area subjects to adolescents with learning disabilities. Journal of Learning Disabilities, 26, 358-383, 398.
Farkas, G. (1996). Human capital or cultural capital? Ethnicity and poverty groups in an urban school district. New York: Aldine de Gruyter.
Hatcher, P. J., Hulme, C., & Ellis, A. W. (1994). Ameliorating early reading failure by integrating the teaching of reading and phonological skills. Child Development, 65, 41-57.
Kavale, K. A., Hirshoren, A., & Forness, S. R. (1998). Meta-analytic validation of the Dunn and Dunn model of learning style preferences: A critique of what was dunn. Learning Disabilities Research & Practice, 13, 75-80.
Mann, V. (1994). Phonological skills and the prediction of early reading problems. In N. C. Jordan & J. Goldsmith-Phillips (Eds.), Learning disabilities: New directions for assessment and intervention (pp. 67-84). Boston, MA: Allyn and Bacon.
Moats, L. C., & Lyon, G. R. (1993). Learning disabilities in the United States: Advocacy, science, and the future of the field. Journal of Learning Disabilities, 26, 282-294.
Nation, K., & Snowling, M. (1997). Assessing reading difficulties: The validity and utility of current measures of reading skill. British Journal of Educational Psychology, 67, 359-370.
Olson, R. K., Rack, J. P., Conners, F. A., DeFries, J. C., & Fulker, D. W. (1991). Genetic etiology of individual differences in reading disability. In L. V. Feagans, E. J. Short, & L. J. Meltzer (Eds.), Subtypes of learning disabilities: Theoretical perspectives and research (pp. 113-135). Hillsdale, NJ: Lawrence Erlbaum Associates.
Rack, J. P., Snowling, M. J., & Olson, R. K. (1992). The nonword reading deficit in developmental dyslexia: A review. Reading Research Quarterly, 27, 28-53.
Salvia, J., & Ysseldyke, J. E. (1995). Assessment (6th edition). Boston, MA: Houghton-Mifflin.
Scarborough, H. S. (1998). Early identification of children at risk for reading disabilities. In B. K. Shapiro, P. J. Accardo, & A. J. Capute (Eds.), Specific reading disability (pp. 75-119). Timonium, MD: York Press.
Stanovich, K. E., Siegel, L. S., Gottardo, A., Chiappe, P., & Sidhu, R. (1997). Subtypes of developmental dyslexia: Differences in phonological and orthographic coding. In B. Blachman (Ed.), Foundations of reading acquisition and dyslexia (pp. 115-141). Mahwah, NJ: Erlbaum.
Sternberg, R. J. (1996). Successful intelligence: How practical and creative intelligence determine success in life. New York: Simon & Schuster.
Sweetland, R. C., & Keyser, D. J. (1993). Tests: A comprehensive reference for assessment in psychology, education, and business (3rd edition). Austin, TX: Pro-Ed.
Torgesen, J. K. (1998). Instructional interventions for children with reading disabilities. In B. K. Shapiro, P. J. Accardo, & A. J. Capute (Eds.), Specific reading disability (pp. 197-220). Timonium, MD: York Press.
Vellutino, F. R. (1979). Dyslexia: Theory and research. Cambridge, MA: MIT Press.
Wagner, R. K., & Torgesen, J. K. (1987). The nature of phonological processing and its causal role in the acquisition of reading skills. Psychological Bulletin, 101, 192-212.
Wolf, M. (1991). Naming speed and reading: The contribution of the cognitive neurosciences. Reading Research Quarterly, 26, 123-141.
Dunn, L., & Dunn, L. (1981). Peabody Picture Vocabulary Test-Revised. Circle Pines, MN: American Guidance Service.
Kaplan, E., Goodglass, H., & Weintraub, S. (1983). Boston Naming Test. Philadelphia: Lea & Febiger.
Lindamood, C. H., & Lindamood, P. C. (1979). Lindamood Auditory Conceptualization Test. Hingham, MA: Teaching Resources Corporation.
Psychological Corporation. (1992). Wechsler Individual Achievement Test. San Antonio, TX: Harcourt Brace Jovanovich.
Torgesen, J. K., & Bryant, B. R. (1994). Test of Phonological Awareness. Austin, TX: Pro-Ed.
Wiederholt, L., & Bryant, B. (1992). Gray Oral Reading Tests-3. Austin, TX: Pro-Ed.
Woodcock, R. (1989). Woodcock Reading Mastery Tests-Revised. Circle Pines, MN: American Guidance Service.
Woodcock, R. W., & Johnson, M. B. (1989). Woodcock-Johnson Tests of Achievement-Revised. Boston, MA: Houghton-Mifflin.
Louise Spear-Swerling, Ph.D. Southern Connecticut State University Co-Author: "Off Track - When Poor Readers Become 'Learning Disabled'" August, 1998