Education Commons^™

Title And Contents- The Computer And The Decision-Making Process, Jane Close Conoley, Terry B. Gutkin, Steven L. Wise

The Computer and the Decision-Making Process

The Computer And The Decision-Making Process

Contents

Preface ix

1. Computer-Assisted Personality Test Interpretation: The Dawn of Discovery
Douglas N. Jackson

Some Preconditions for Valid Computer- Assisted Test Interpretation 1
The Dawn of Discovery 6
References 9

2. Psychodiagnostic Computing: From Interpretive Programs to Expert Systems 11
Marley W Watkins and Paul A. McDermott

Development of Administration and Interpretation Programs 11
Novel Administration and Interpretation Programs 14
Computerized Interpretation Systems 16
A Psychoeducational Diagnostic Model 19
A Computerized Psychoeducational Diagnostic System 23
Summary 36
References 37

3. Assessment of Validity in Computer-Based Test Interpretations
Kevin L. Moreland

Some History 44 …

Preface, Jane Close Conoley

The Computer and the Decision-Making Process

This volume in the Buros-Nebraska Series on Testing and Measurement provides state-of-the-art contributions concerning the interface between computer technology and traditional psychometrics. The volume title, Computers and the Decision-Making Process, describes both reality and potential in a field that provides a dizzying array of promises and problems to be pursued and be solved.

This volume like the previous ones in our series reflects papers given at the annual Buros-Nebraska Symposium on Testing and Measurement and those especially commissioned for the book. Each of the contributors has a special expertise to examine the complex issues raised by the addition of …

1. Computer-Assisted Personal Ity Test Interpretation: The Dawn Of Discovery, Douglas N. Jackson

The Computer and the Decision-Making Process

My aim in this chapter is to outline some of the substantive and psychometric bases on which we can build a science of assessment that takes advantage of the enormous potential inherent in the digital computer and in artificial intelligence. Some of these foundations are within the traditions of classical assessment. But others represent urgently needed areas of explication and research.

It is my view, in the tradition of Cronbach (1954), that developers of computer software for testing should listen to what psychometricians say, and, as well, psychometricians should be sensitive to new research ideas waiting to be solved that …

2. Psychodiagnostic Computing: From Interpretive Programs To Expert Systems, Marley W. Watkins, Paul A. Mcdermott

The Computer and the Decision-Making Process

As amply demonstrated by the chapters in this volume, computer applications have pervaded all aspects of psychological practice. Although thought by some to be relatively new (Nolen & Spencer, 1986), semiautomatic scoring of the Strong Vocational Interest Blank was accomplished more than 50 years ago (Campbell, 1968) and systems of computer-based test interpretation have been operational for 25 years (Fowler, 1985).

DEVELOPMENT OF ADMINISTRATION AND INTERPRETATION PROGRAMS

Early automated programs typically focused upon the scoring or interpretation of a single psychological test. Most frequently, that test was the Minnesota Multiphasic Personality Inventory (Fowler, 1985) but the Rorschach was interpreted as …

3. Assessment Of Val Id Ity In Computer-Based Test Interpretations, Kevin L. Moreland

The Computer and the Decision-Making Process

The use of computers to interpret psychological tests is a "hot" topic, both within psychology and without. It is hot in the sense of giving rise to an increasing number of books and articles (e.g., Butcher, 1985, 1987; Eyde, 1987; Krug, 1987). It is hot in the sense of giving rise to an ever-increasing number of business enterprises (compare any recent APA Monitor with an issue from 1981). It is hot in the sense of capturing the attention of the news media (e.g., Petterson, 1983). And it is hot in the sense of giving rise to increasing controversy within psychology …

4. The Validity Of Computerbased Test Interpretations Of The Mmpi, Lorraine D. Eyde, Dennis M. Kowal, Francis J. Fishburne Jr.

The Computer and the Decision-Making Process

With advances in computer technology, computer-based test interpretations (CBTI), first developed in the early 1960s (Fowler, 1985), have proliferated (Eyde & Kowal, 1987). CBTIs have been developed and marketed for a variety of tests used in clinical, counseling, educational, and employment settings. The largest number of commercial CBTI systems are available for the Minnesota Multiphasic Personality Inventory (MMPI; Krug, 1987), the most widely used inventory of its kind in the world, which has a continuously growing literature of more than 8,000 books and articles (Holden, 1986; Lanyon, 1984).

According to Harris:
CBTI refers to the automation of a set of …

5. Use Of Computer Technology In Behavioral Assessments, Thomas R. Kratochwill, Elizabeth J. Doll, W. Patrick Dickson

The Computer and the Decision-Making Process

Major developments in the behavioral assessment field have occurred over the past decade (e.g., Barlow, 1981 ; Ciminero, Calhoun, & Adams , 1986; Haynes & Wilson, 1979; Mash & Terdal, 1988a). The use of computer technology by behavioral assessors has occurred, but this is a relatively recent development (Kratochwill, Doll, & Dickson, 1986; Romanczyk, 1986). Consider, for example, that behavioral assessment texts include little discussion of computer applications and many articles restrict discussion of behavioral assessment to observational measures (see Cone & Hawkins, 1977, for an exception). In psychology and education, issues of journals have been devoted to computer applications …

6. The Use Of The Computer In The Practice Of Industrial/ Organizational Psychology, Lyle F. Schoenfeldt, Jorge L. Mendoza

The Computer and the Decision-Making Process

The rapid proliferation of computer technology, in the form of mainframe computers, networks of interconnected machines, and stand-alone personal computers, is having a profound effect on many areas of life. As a result of the spread of computer equipment to offices, homes, and educational institutions; the variety of software applications has grown at an unprecedented rate. With this as background, it should be no surprise that computers have assumed an increasing role in professional practice, including applications in providing services in the area of industrial and organizational psychology.

Industrial-organizational psychologists function in a variety of settings, but primarily provide human …

8. Issues In Intelligent Computer-Assisted Instruction: Eval Uation And Measurement, Harold F. O'Neil Jr., Eva L. Baker

The Computer and the Decision-Making Process

In this chapter we plan to explore two issues in the field of intelligent computer assisted instruction (ICAI) that we feel offer opportunities to advance the state of the art. These issues are evaluation of ICAI systems and the use of the underlying technology in ICAI systems to develop tests. For each issue we will provide a theoretical context, discuss key constructs, provide a brief window to the appropriate literature, suggest methodological solutions and conclude with a concrete example of the feasibility of the solution from our own research.

INTELLIGENT COMPUTER-ASSISTED INSTRUCTION (ICAI)

ICAI is the application of artificial intelligence …

Author Index

The Computer and the Decision-Making Process

Author Index (10 pages)

A-Z

A

Acker, N. E., 130, 154
Adair, F. L., 51, 52, 57,68,87,89, 114
Adams, H. E., 125, 151
Adams, K. M., 13,37,51,59, 68
Agras, W. S., 136, 137, 151
Alessi, S. M., 209, 221
Algozzine, B., 19,37
Aikin, M. C., 204, 219, 221
Allen, B. A. , 12,41
Allred, L. J., 249, 253
Allusisi, E. A., 156, 174
Alpert, D., 149, 150
Altman, H., 49, 71
Anastasi, A., 77, 114
Andl, R., 129, 151, 239, 242
Anderson, B. N., 52, 68
Anderson, C. L., 201, 224
Anderson, R. J., 201, 221
Anderson, T., 126, 142, …

10. Guidelines For Computer Testing, Bert F. Green

The Computer and the Decision-Making Process

Testing by computer is big business. Many companies are offering software enabling a psychologist to test a client by seating him or her at a computer terminal and pressing Return. The software presents the instructions on the screen, guides the test taker through some sample items to see if the instructions are understood, and then presents the test, automatically recording the responses. After one or more tests have been completed, the equipment scores the responses, and delivers test scores. But it doesn't stop there. It then continues by printing out a complete test interpretation in fairly well-constructed narrative prose. The …

So Much Content, So Little Time, Marilla D. Svinicki

Professional and Organizational Development Network in Higher Education: Archives

The universal complaint of faculty is that there is too much content to cover in the time allotted. Rather than complaining, perhaps we should re- examine how we go about choosing the content to include in a course. We might find we have more than we really need.

Countering Common Misbeliefs About Student Evaluation Of Teaching, Robert Boice

Professional and Organizational Development Network in Higher Education: Archives

In spite of all evidence to the contrary some people continue to believe that student evaluation of teaching is “nothing but a popularity contest.” How much more useful it is to recognize what the students have to offer in the way of feedback on teaching.

Teaching: Beliefs And Behaviors, Robert J. Menges

Professional and Organizational Development Network in Higher Education: Archives

Our beliefs about learning shape the behaviors of our teaching. We need to be aware of what they are and how they influence our actions.

Collaborative Learning: Reframing The Classroom, Jean Macgregor

Professional and Organizational Development Network in Higher Education: Archives

The method of collaborative learning goes far beyond a change in teaching methodology. It is a change in the whole relationship between learners and the environment.

Forward To Aristotle: Teaching As The Highest Form Of Understanding, Russell Edgerton

Professional and Organizational Development Network in Higher Education: Archives

There is more to teaching than a mere grasp of content. The act of teaching itself is a complex and fascinating experience which goes beyond simply knowing the subject and talking about it.

Title And Contents, Jane Close Conoley, James Y. Mitchell Jr, Steven L. Wise, Barbara S. Plake

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

Buros-Nebraska Symposium on Measurement & Testing

Contents

Preface vii

1. Face Validity: Siren Song for Teacher-Testers 1
W. James Popham

2. Teacher Performance Assessments: A New Kind of Teacher Examination 15
Edward H. Haertel

3. Improving Teaching Through the Assessment Process 37
Donald M. Medley

4. Assessing the Quality of Teacher Assessment Tests 77
William A. Mehrens

5. Teacher Evaluation in the Organizational Context 137
Linda Darling-Hammond

6. Measuring Performance in Teacher Assessment 183
Richard J. Stiggins

7. Legal and Professional Issues in Teacher Certification Testing: A Psychometric Snark Hunt 209
George F. Madaus

8. Limitations of Using Student Achievement …

Preface, Barbara S. Plake

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The Buros-Nebraska annual Symposia on Measurement and Testing are aimed at providing a forum for discussing important issues in the field of measurement and testing. The topic for the 1987 Symposium was "Assessment of Teaching: Purposes, Practices, and Implications for the Profession." This topic was selected because of the current interest in developing, designing, and implementing accountability programs for teaching that are present in many states and education programs. The complex nature of teaching, combined with the unique measurement issues for assessing outcomes in the teaching context, provided the basis for identifying the topic of teacher assessment for the 1987 …

1. Face Validity: Siren Song For Teacher Testers, W. James Popham

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The sirens of Greek mythology were a seductive set of women who, by singing melodies that apparently topped even those of Diana Ross and the Supremes, could lure mesmerized men to their doom. Greek mythology, it is clear, was solidly sexist, for the sirens used their supernatural singing talents to entice only unsuspecting males into trouble. Gender-equity considerations were conspicuously absent from the forays of Greek fablemakers. Sexism aside, however, it is certain that the sirens of yesteryear knew how to sing some truly enticing tunes.

FACE VALIDITY'S ALLURE

In today's current frenzy to develop teacher assessment devices that tap …

10. Appraisal: The Teachers' Perspective, Peg Shafer

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The Discovery Channel on TV has some terrific programming. Not too long ago I watched a scientist explain why it is that the position of a stranded boat will not change even when it is tossed around by giant storms. The theory is that waves move across the water rather than the water itself moving. The wave moves along in a circular motion, propelled by big or small forces depending on what started it in the first place. As long as nothing interrupts the roll of the wave, it will pass and leave everything as it was before.

At the …

9. Teaching Assessment: The Administrator's Perspective, John R. Hoyle

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

School administrators, especially principals, are under great pressure to insure high levels of teacher competence. Because the school effectiveness research has demonstrated convincingly that effective schools begin with effective principals, Peterson and Finn (1985) drew a less than surprising conclusion by stating that "Practically never does one encounter a good school with a bad principal" (p. 42). A less pedantic east Texas superintendent put it this way, "Bad principals are like fish; you either can 'em or smell 'em for a long time." It is in the complex area of teaching assessment or teacher evaluation that principals draw the most …

2. Teacher-Performance Assessments: A New Kind Of Teacher Examination, Edward H. Haertel

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

During the last week of July 1987,20 fourth- and fifth-grade teachers spent four days at a simulated assessment center in an elementary school. Each teacher completed 10 performance exercises on the teaching of equivalent fractions. The following week, 20 high school teachers of United States history spent four days completing a like number of exercises on the American revolution and the formation of the new government. These field tests were the culmination of over a year's work by the Teacher Assessment Project (TAP), sponsored by the Carnegie Corporation of New York, under the direction of Professor Lee S. Shulman at …

6. Measuring Performance In Teacher Assessment, Richard J. Stiggins

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

In performance assessment, the examinee is called upon to display behaviors or produce products which an observer evaluates in terms of pre-specified performance standards. This kind of measurement plays a major role in many school evaluation contexts, including both student and teacher evaluation. The observation and evaluation of a teacher's performance in the classroom represents one of many excellent sources of information on teacher capabilities and the effectiveness of instruction. The purpose of this chapter is to review the various ways performance assessment methodology can serve in teacher-assessment contexts.

The review begins with a summary of the range of settings …

3. Improving Teaching Through The Assessment Process, Donald M. Medley

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The first opportunity to use teacher evaluation to improve teaching arises when a student applies for admission into an undergraduate teacher preparation program. At this time it is the responsibility of the program faculty to determine whether each candidate possesses those abilities and other personal characteristics that every teacher needs, but cannot expect to acquire in such a program, and to deny admission to those who lack one or more of them. The second opportunity arises when the student has completed the program. At this time it is the responsibility of the state certification agency to find out whether each …

8. Limitations Of Using Student-Achievement Data For Career-Ladder Promotions And Merit-Pay Decisions, Ronald A. Berk

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

A study of U.S. school districts conducted 70 years ago reported that 48% of the districts sampled used merit pay (Evendon, 1918). Since then, the quantity as well as quality of teacher-compensation systems has fluctuated markedly (for details, see Cohen & Murnane, 1985; Murnane & Cohen, 1986; Porwoll, 1979). At present, 29 states are implementing large-scale teacher-incentive programs (a.k.a. career ladder, merit pay, pay for performance), funding local plans, piloting testing models, or using state board of education or legislative mandates to develop programs for teachers and administrators (Southern Regional Education Board, 1986) The status of these programs is summarized …

5. Teacher Evaluation In The Organizational Context, Linda Darling-Hammond

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

Personnel evaluation in an organization provides a powerful metaphor for what is valued in the organization, how roles are construed, and which goals have de facto priority in the management of organizational affairs. The importance attached to this function says much about the organization's relationship to its clients, as well as the relationships among organizational members. The same can be said about the importance of evaluation in an occupation whose members share a common service mission. Indeed, evaluation plays a particularly critical role in an occupation that claims to be a profession. This chapter explores the role of teacher evaluation …

7. Legal And Professional Issues In Teacher-Certification Testing: A Psychometric Snark Hunt, George F. Madaus

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

The validation of teacher-certification tests has become a snark hunt. The snark pursued by test contractors however, is not nearly as elusive, nor fanciful as Carroll's shadowy beast. Their snark is a "legally defensible" test. Test contractors are-merely from nervousness (not from goodwill)-marching along shoulder to shoulder with jurists. I will argue that hunting the shadowy snark through the dark wood of the court room cheapens the conceptualization of validity and delimits the conduct of validation studies to minimalist exercises designed to obtain a positive result.

I start from the basic premise that the most important feature of any test …

4. Assessing The Quality Of Teacher Assessment Tests, William A. Mehrens

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

This chapter discusses some of the types of evidences that are appropriate for assessing the quality of teacher-licensure tests. Licensure tests are used to make dichotomous decisions, so reliability estimates of the consistency of decisions are needed. Because the inference of interest has to do with the minimum competency necessary to prevent harm from coming to the clients, it is argued that content validity is the type of validity evidence most appropriate for licensure tests. However, evidences for criterion-related validity, construct validity and "curricular validity" are also discussed. The issue of whether the cut score on a licensure test should …

Author Index

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

Author Index (8 pages)

A-Z

A

Abeles, S., 18,35
Aburto, S., 22, 34
Acheson, K. A., 311, 326
Adams, N. E., 173,176
Adams, P. T., 47, 75
Aickin, M., 217, 257
Airasian, P. W., 283, 303
Alabama State Board of Education, 81, 129
American Educational Research Association, 8, 14, 28, 34, 83, 85, 86, 88, 97, 98, 102, 105, 106, 107, 109, 119, 125, 126, 127,129,227,246,247,256,267,268, 269, 277, 282, 286, 298, 300, 359, 375
American Psychological Association, 8, 14, 28,34,83, 85, 86, 88, 92, 97, 98 , 102, 105, 106, 107, 109, 119, 125, 126, 127, 129,227,246,247,256,267,268,269, 277,282, 286, 298, …

11. The Assessment Of Teacher Assessment: Concluding Thoughts And Some Lingering Questions, James V. Mitchell Jr.

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

In a recent update on teacher testing practices across the United States, Rudner (1988) reported that 44 states have developed teacher-certification-testing programs, with 26 states currently testing prospective teachers as a certification requirement and another 18 states scheduled to implement such programs in the near future. It is obvious that teacher testing has become a very extended endeavor- It has also stimulated extended debate.

It was in acknowledgment of the importance of this extended teacher testing and associated debate that the Advisory Committee of the Buros Institute of Mental Measurements decided to devote its 1987 annual symposium to the topic …