Developing and Validating Multiple-choice Test Items

Thomas Haladyna has written "...a comprehensive treatment of the development of multiple-choice (MC) test items and the study of item responses for the improvement or continued use of these items." (p. vii). The author graciously acknowledges that "...many testing specialists and educators have contributed to this book..." but he has reviewed these contributions and synthesized them into this well-written guide. The book overviews basic concepts of test validity and how to interpret test scores responsibly. Haladyna acknowledges testing standards developed by the American Educational Research Association (AERA), the National Council on Measurement in Education (NCME), and the American Psychological Association. Test validity is extended to the level of individual test questions ("items"). The validity of test items is established in two ways. First, appropriate procedures must be followed as they are written and reviewed. Then responses of test takers must be analyzed to identify and correct any remaining problems. Haladyna lays the foundation for item writing in Chapter 2 by discussing the mental processes that test developers hope to elicit. They should not only design items with appropriate content, but consider the cognitive demand of each item. Chapter 3 reviews the basic MC format of a question, a set of response options and a scoring protocol. Haladyna distinguishes between high-inference (abstract) and low-inference (concrete) items and outlines the appropriate role of each. (See the summary table on p.45.) "Not only must the item writer understand content measured by the item, but must determine whether the cognitive demand will involve recall, understanding or application." (p. 65). This is complicated by the fundamental inference problem of testing: We can never be sure whether an item elicits an intended complex mental process or simple recall of a memorized answer. The chapter closes with recommendations for choosing between MC items and open-ended, constructed response (CR) items. The book's second section presents MC item formats. Chapter 4 describes eight formats, including the familiar four-option MC and the much-dreaded true-false item. Chapter 5 reminds readers that MC development produces casualties, with only 50% of MC items crossing the finish line. Haladyna outlines practices for creating test specifications, training item writers, and editing their work. He also cites research showing that these practices produce better test items, lest we be tempted to skip a step. Chapter 6 contains examples of well-written items and ill-conceived items with interesting and instructive flaws. Chapter 7 closes the section with approaches that increase efficiency and standardization of item pools by using structured specifications for item writing. The next two chapters contain Haladyna's central messages. "The quality of test items depends on two complementary activities, the item review procedures featured in Ch