Educational Assessment, Evaluation, and Research Commons^™

1. Face Validity: Siren Song For Teacher Testers, W. James Popham

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The sirens of Greek mythology were a seductive set of women who, by singing melodies that apparently topped even those of Diana Ross and the Supremes, could lure mesmerized men to their doom. Greek mythology, it is clear, was solidly sexist, for the sirens used their supernatural singing talents to entice only unsuspecting males into trouble. Gender-equity considerations were conspicuously absent from the forays of Greek fablemakers. Sexism aside, however, it is certain that the sirens of yesteryear knew how to sing some truly enticing tunes.

FACE VALIDITY'S ALLURE

In today's current frenzy to develop teacher assessment devices that tap …

10. Appraisal: The Teachers' Perspective, Peg Shafer

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The Discovery Channel on TV has some terrific programming. Not too long ago I watched a scientist explain why it is that the position of a stranded boat will not change even when it is tossed around by giant storms. The theory is that waves move across the water rather than the water itself moving. The wave moves along in a circular motion, propelled by big or small forces depending on what started it in the first place. As long as nothing interrupts the roll of the wave, it will pass and leave everything as it was before.

At the …

9. Teaching Assessment: The Administrator's Perspective, John R. Hoyle

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

School administrators, especially principals, are under great pressure to insure high levels of teacher competence. Because the school effectiveness research has demonstrated convincingly that effective schools begin with effective principals, Peterson and Finn (1985) drew a less than surprising conclusion by stating that "Practically never does one encounter a good school with a bad principal" (p. 42). A less pedantic east Texas superintendent put it this way, "Bad principals are like fish; you either can 'em or smell 'em for a long time." It is in the complex area of teaching assessment or teacher evaluation that principals draw the most …

6. Measuring Performance In Teacher Assessment, Richard J. Stiggins

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

In performance assessment, the examinee is called upon to display behaviors or produce products which an observer evaluates in terms of pre-specified performance standards. This kind of measurement plays a major role in many school evaluation contexts, including both student and teacher evaluation. The observation and evaluation of a teacher's performance in the classroom represents one of many excellent sources of information on teacher capabilities and the effectiveness of instruction. The purpose of this chapter is to review the various ways performance assessment methodology can serve in teacher-assessment contexts.

The review begins with a summary of the range of settings …

3. Improving Teaching Through The Assessment Process, Donald M. Medley

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The first opportunity to use teacher evaluation to improve teaching arises when a student applies for admission into an undergraduate teacher preparation program. At this time it is the responsibility of the program faculty to determine whether each candidate possesses those abilities and other personal characteristics that every teacher needs, but cannot expect to acquire in such a program, and to deny admission to those who lack one or more of them. The second opportunity arises when the student has completed the program. At this time it is the responsibility of the state certification agency to find out whether each …

8. Limitations Of Using Student-Achievement Data For Career-Ladder Promotions And Merit-Pay Decisions, Ronald A. Berk

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

A study of U.S. school districts conducted 70 years ago reported that 48% of the districts sampled used merit pay (Evendon, 1918). Since then, the quantity as well as quality of teacher-compensation systems has fluctuated markedly (for details, see Cohen & Murnane, 1985; Murnane & Cohen, 1986; Porwoll, 1979). At present, 29 states are implementing large-scale teacher-incentive programs (a.k.a. career ladder, merit pay, pay for performance), funding local plans, piloting testing models, or using state board of education or legislative mandates to develop programs for teachers and administrators (Southern Regional Education Board, 1986) The status of these programs is summarized …

4. Assessing The Quality Of Teacher Assessment Tests, William A. Mehrens

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

This chapter discusses some of the types of evidences that are appropriate for assessing the quality of teacher-licensure tests. Licensure tests are used to make dichotomous decisions, so reliability estimates of the consistency of decisions are needed. Because the inference of interest has to do with the minimum competency necessary to prevent harm from coming to the clients, it is argued that content validity is the type of validity evidence most appropriate for licensure tests. However, evidences for criterion-related validity, construct validity and "curricular validity" are also discussed. The issue of whether the cut score on a licensure test should …

11. The Assessment Of Teacher Assessment: Concluding Thoughts And Some Lingering Questions, James V. Mitchell Jr.

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

In a recent update on teacher testing practices across the United States, Rudner (1988) reported that 44 states have developed teacher-certification-testing programs, with 26 states currently testing prospective teachers as a certification requirement and another 18 states scheduled to implement such programs in the near future. It is obvious that teacher testing has become a very extended endeavor- It has also stimulated extended debate.

It was in acknowledgment of the importance of this extended teacher testing and associated debate that the Advisory Committee of the Buros Institute of Mental Measurements decided to devote its 1987 annual symposium to the topic …

Rethinking What It Means To Be A Scholar, R. Eugene Rice

Professional and Organizational Development Network in Higher Education: Archives

A broader and more accurate definition of scholarship also includes the integration of knowledge, the application of knowledge, and the representation and dissemination of knowledge-teaching.

Teaching In Action: Criteria For Effective Practice, Ronald Smith, Fred Schwartz

Professional and Organizational Development Network in Higher Education: Archives

This essay describes ways through which teachers enhance their teaching art within the process of teaching itself.

The Meaning Of College Grades, James Eison

Professional and Organizational Development Network in Higher Education: Archives

The often confusing and inconsistent character of academic grades in higher education can be greatly reduced by suggestions offered in this essay.

Why Professors Don't Change, Loren Ekroth

Professional and Organizational Development Network in Higher Education: Archives

Even if professors wanted to change their teaching methods, the ways they define themselves and stabilizing aspects of their academic situations often resist their attempts to change.

The Challenge Of Teaching The Introductory-Level Course, Delivee L. Wright

Professional and Organizational Development Network in Higher Education: Archives

Introductory-level courses usually offer many more and greater challenges than do advanced courses, and it is appropriate that they be delivered by the most effective instructors.

Critical Thinking By Design, Joanne Gainen Kurfiss

Professional and Organizational Development Network in Higher Education: Archives

This essay offers guidelines for increasing the critical thinking potential of courses in the disciplines in higher education.

Title And Contents, Jane Close Conoley, James Y. Mitchell Jr, Steven L. Wise, Barbara S. Plake

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

Buros-Nebraska Symposium on Measurement & Testing

Contents

Preface vii

1. Face Validity: Siren Song for Teacher-Testers 1
W. James Popham

2. Teacher Performance Assessments: A New Kind of Teacher Examination 15
Edward H. Haertel

3. Improving Teaching Through the Assessment Process 37
Donald M. Medley

4. Assessing the Quality of Teacher Assessment Tests 77
William A. Mehrens

5. Teacher Evaluation in the Organizational Context 137
Linda Darling-Hammond

6. Measuring Performance in Teacher Assessment 183
Richard J. Stiggins

7. Legal and Professional Issues in Teacher Certification Testing: A Psychometric Snark Hunt 209
George F. Madaus

8. Limitations of Using Student Achievement …

7. Legal And Professional Issues In Teacher-Certification Testing: A Psychometric Snark Hunt, George F. Madaus

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The validation of teacher-certification tests has become a snark hunt. The snark pursued by test contractors however, is not nearly as elusive, nor fanciful as Carroll's shadowy beast. Their snark is a "legally defensible" test. Test contractors are-merely from nervousness (not from goodwill)-marching along shoulder to shoulder with jurists. I will argue that hunting the shadowy snark through the dark wood of the court room cheapens the conceptualization of validity and delimits the conduct of validation studies to minimalist exercises designed to obtain a positive result.

I start from the basic premise that the most important feature of any test …

Author Index

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

Author Index (8 pages)

A-Z

A

Abeles, S., 18,35
Aburto, S., 22, 34
Acheson, K. A., 311, 326
Adams, N. E., 173,176
Adams, P. T., 47, 75
Aickin, M., 217, 257
Airasian, P. W., 283, 303
Alabama State Board of Education, 81, 129
American Educational Research Association, 8, 14, 28, 34, 83, 85, 86, 88, 97, 98, 102, 105, 106, 107, 109, 119, 125, 126, 127,129,227,246,247,256,267,268, 269, 277, 282, 286, 298, 300, 359, 375
American Psychological Association, 8, 14, 28,34,83, 85, 86, 88, 92, 97, 98 , 102, 105, 106, 107, 109, 119, 125, 126, 127, 129,227,246,247,256,267,268,269, 277,282, 286, 298, …

Subject Index

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

Subject Index (6 pages)

A-Z

A

AASPB,114
Academic Learning Time (ALT), 324
Administrators, 316-317, 321, 324 See also Principals
Admissions Research, 45-46
Adverse Impact, 217-225, 227
Affective Variables, 355-356
Albemarle Paper Co. v. Moody, 217, 232
Allen v. The Alabama State Board of Education, 129, 214-215, 227, 232, 234-236, 238, 241n, 242, 248
Applied Testing, 242-245, 255
Appraisal Reform, 330, 341
Arkansas Educational Skills Assessment (AESA), 11

...

WY

Washington v. Davis, 228-231
Wesman Personnel Classification Test (WPCT),104
Wisdom, 41, 48
Wisdom of Practice, 23
Withitness, 51, 61
Work Environment Variables, 272
York v. Mobile, 238n

2. Teacher-Performance Assessments: A New Kind Of Teacher Examination, Edward H. Haertel

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

During the last week of July 1987,20 fourth- and fifth-grade teachers spent four days at a simulated assessment center in an elementary school. Each teacher completed 10 performance exercises on the teaching of equivalent fractions. The following week, 20 high school teachers of United States history spent four days completing a like number of exercises on the American revolution and the formation of the new government. These field tests were the culmination of over a year's work by the Teacher Assessment Project (TAP), sponsored by the Carnegie Corporation of New York, under the direction of Professor Lee S. Shulman at …

5. Teacher Evaluation In The Organizational Context, Linda Darling-Hammond

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

Personnel evaluation in an organization provides a powerful metaphor for what is valued in the organization, how roles are construed, and which goals have de facto priority in the management of organizational affairs. The importance attached to this function says much about the organization's relationship to its clients, as well as the relationships among organizational members. The same can be said about the importance of evaluation in an occupation whose members share a common service mission. Indeed, evaluation plays a particularly critical role in an occupation that claims to be a profession. This chapter explores the role of teacher evaluation …

Assessment-Of Teaching: Purposes, Practices, And Implications For The Profession- Complete Work, Jane Close Conoley, James Y. Mitchell Jr., Steven L. Wise, Barbara S. Plake

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The Buros-Nebraska annual Symposia on Measurement and Testing are aimed at providing a forum for discussing important issues in the field of measurement and testing. The topic for the 1987 Symposium was "Assessment of Teaching: Purposes, Practices, and Implications for the Profession." This topic was selected because of the current interest in developing, designing, and implementing accountability programs for teaching that are present in many states and education programs. The complex nature of teaching, combined with the unique measurement issues for assessing outcomes in the teaching context, provided the basis for identifying the topic of teacher assessment for the 1987 …

Section I- The Cognitive Psychometric Connection

The Influence of Cognitive Psychology on Testing

Section I- The Cognitive Psychometric Connection

Section Ii- Cognitive Approaches To Psychometric Issues: Applications

The Influence of Cognitive Psychology on Testing

Section II- COGNITIVE APPROACHES TO PSYCHOMETRIC ISSUES: APPLICATIONS

Section Iii- Methodological Issues

The Influence of Cognitive Psychology on Testing

Section III- METHODOLOGICAL ISSUES

Title Page And Contents- The Influence Of Cognitive Psychology On Testing, Jane Close Conoley, Royce R. Ronning, John A. Glover, Joseph C. Witt

The Influence of Cognitive Psychology on Testing

The Influence of Cognitive Psychology on Testing

Contents

Foreword ix

1. Introduction: The Implications of Cognitive Psychology for Testing...........1

PART I: THE COGNITIVE-PSYCHOMETRIC CONNECTION

2. Science, Technology, and Intelligence...................11

3. Toward a Cognitive Theory for the Measurement of Achievement ...................41

4. The g Beyond Factor Analysis.........................87

PART II: COGNITIVE APPROACHES TO PSYCHOMETRIC ISSUES: APPLICATIONS

5. The Assessment of Cognitive Factors in Academic Abilities...................145

6. Theoretical Implications from Protocol Analysis on Testing and Measurement...191

PART III: METHODOLOGICAL ISSUES

7. Structure and Process in Cognitive Psychology Using Multidimensional Scaling and Related Techniques..................229

8. New Perspectives in the Analysis of Abilities.....................267

Author Index …

Foreword- The Influence Of Cognitive Psychology On Testing, James V. Mitchell Jr.

The Influence of Cognitive Psychology on Testing

For over 40 years Oscar K. Buros was Director of the Buros Institute of Mental Measurements and Editor of the Mental Measurements Yearbooks. He was a crusader, and he devoted his entire career to his crusade. He was a crusader for better tests and the more effective selection and use of tests, and he used the Mental Measurements Yearbooks as the principal instrument in this crusade. Buros passed away in 1978, and his widow, Luella Buros, worked tirelessly to find a new home for the Institute. As a result of her efforts the Institute was relocated at the University …

2. Science, Technology, And Intelligence, Earl Hunt

The Influence of Cognitive Psychology on Testing

The intelligence test has been cited as psychology's most important technological contribution to society. Whether this is good or ill can be debated (Eysenck, 1979; Gould, 1981; Herrnstein, 1971; Kamin, 1974). Certain facts are not really subject to debate . Psychologists can and have developed "standardized interviews" that, on a population basis, provide a cost effective technique for personnel classification in industrial, military, and some government settings. However, the tests are very far from perfect indicators. Validity coefficients between tests and performance ratings typically range in the .3 to .5 range (i.e. , from 10 to 25% of the variance …

6. Theoretical Implications From Protocol Analysis On Testing And Measurement, K. Anders Ericsson

The Influence of Cognitive Psychology on Testing

One of the goals of psychology has always been to describe, understand, and measure individual differences. The diversity of human behavior makes it particularly challenging to seek to identify general and stable underlying elements that correspond to systematic individual differences . A major problem in the efforts to identify such elements is that the elements cannot be observed directly. The primary method has been to use the current psychological theory to develop procedures to measure such hypothetical elements. In this chapter I present a new theoretic framework, based on verbal reports from subjects, for identifying and measuring individual differences. I …

7. Structure And Process In Cognitive Psychology Using Multidimensional Scaling And Related Techniques, Edward J. Shoben, Brian H. Ross

The Influence of Cognitive Psychology on Testing

INTRODUCTION

The goal of cognitive psychology is to provide a general understanding of human cognitive processes through the development of general, formal models of cognition. Although it is clearly true that some areas (such as memory) have been more highly developed than others, it is undeniable that cognitive psychology has witnessed a proliferation of models in the past decade. Perhaps researchers are finding it increasingly difficult to discriminate among competing memory models because the constraints are so weak. One possibility that will be explored in this chapter is the prospect of using multidimensional scaling (MDS) and related procedures as a …

3. Toward A Cognitive Theory For The Measu Rement Of Achievement, Robert Glaser, Alan Lesgold, Susanne Lajoie

The Influence of Cognitive Psychology on Testing

INTRODUCTION

Given the demands for higher levels of learning in our schools and the press for education in the skilled trades, the professions, and the sciences, we must develop more powerful and specific methods for assessing achievement. We need forms of assessment that educators can use to improve educational practice and to diagnose individual progress by monitoring the outcomes of learning and training. Compared to the well-developed technology for aptitude measurement and selection testing, however, the measurement of achievement and diagnosis of learning problems is underdeveloped. This is because the correlational models that support prediction are insufficient for the task …