Educational Assessment, Evaluation, and Research Commons^™

The Phenomenon Of Large Classes And Practical Suggestions For Teaching Them, Frank Gillespie

Professional and Organizational Development Network in Higher Education: Archives

Large classes are a phenomenon of higher education today. However, large does not preclude providing an effective teaching and learning environment. The environment can be analyzed, good teaching can be modeled, and practical strategies offered.

Challenges In Using Technology For The Improvement Of Undergraduate Education, G. Roger Sell

Professional and Organizational Development Network in Higher Education: Archives

We may believe that access to more information will improve undergraduate education, but students conceptual foundations and critical thinking skills are essential for using information technologies and constructing knowledge.

Teaching And Values: What Values Will We Take Into The 21st Century?, Kathleen Mcgrory

Professional and Organizational Development Network in Higher Education: Archives

Our values drive our decision-making about a range of everyday concerns as professionals. This essay identifies some current values being addressed in scholarly inquiry and attempts to predict the role of values inquiry in curriculum and teaching in the 21st century.

The American Professoriate In Transition, Ronald D. Simpson, Thomas G. Dyer

Professional and Organizational Development Network in Higher Education: Archives

After a brief review of the history of the American professoriate, major trends for the future, the lives and work of today’s faculty members will be discussed in terms of forces influencing the direction of higher education in our society.

7. Basic Psychometric Issues In Licensure Testing, Howard W. Stoker, James C. Impara

Licensure Testing: Purposes, Procedures, and Practices

INTRODUCTION

The number of people in the United States who carry some responsibility for the writing of examination questions and the construction of tests is unknown. In the Preface to The Construction and Use of Achievement Examinations, published by the American Council on Education in 1936, the authors indicated that the number probably exceeded a million. That number has certainly grown in the past 60 years. Questions are posed to students by teachers at all levels of education; the Armed Forces have people whose job it is to construct tests which are used in the promotion of personnel; over …

4. Practice Analysis: Building The Foundation For Validity, Joan E. Knapp, Lenora G. Knapp

Licensure Testing: Purposes, Procedures, and Practices

INTRODUCTION

A review of the literature associated with job analysis reveals two extremes of opinion as represented by the following provocative quotes:

Historically job analysis has been a relatively soporific area of industrial and organizational psychology, characterized by neither heated controversy nor prominent visibility in the research literature. (Harvey, 1991, p. 71)

Validation was once a priestly mystery, a ritual behind the scenes with the professional elite as witness and judge. Today it is a public spectacle combining the attraction of chess and mud wrestling. (Cronbach, 1988, p. 3)

Both our evaluation of practice analysis research and our professional experience …

5. Systematic Item Writing And Test Construction, Anthony Laduca, Steven M. Downing, Thomas R. Henzel

Licensure Testing: Purposes, Procedures, and Practices

Standardized objective testing remains the most popular mode of licensure testing. Even where other types of tests are incorporated, it is often the case that they are provided as complimentary to standardized, multiple-choice (MC) tests. Moreover, scoring theories and standard-setting procedures have been developed over the years in the context of standardized MC testing. At the same time, critics have pointed to limitations of contemporary MC testing practices, including lack of fidelity to real-life challenges and emphasis on recall of factual minutiae. In our view, testing professionals should make conscientious attempts to modify test development procedures so as to address …

3. Policy Issues With Psychometric Implications, Michael Rosenfeld, Richard F. J. Tannenbaum, Scott Wesley

Licensure Testing: Purposes, Procedures, and Practices

Testing candidates with disabilities, testing repeaters, and coaching involve issues of fairness, the validity of the inferences made from test scores, and protection of the public. Licensing boards must develop policies to deal with each of these issues. It is interesting to note that although all three are of concern to licensing agencies, little of the research on these topics has been conducted in licensure settings. This chapter discusses the results of research conducted on each topic, considers the psychometric implications for policy of each, and suggests steps licensing boards can take when formulating policy.

TESTING CANDIDATES WITH DISABILITIES IN …

8. Item Banking, Betty A. Bergstrom, Richard C. Gershon

Licensure Testing: Purposes, Procedures, and Practices

Item banks developed by licensure agencies range from a collection of items stored on index cards to highly sophisticated electronic databases. Regardless of the storage mechanism, most banks contain items that have been organized and referenced according to procedures established by the licensure agency. This chapter outlines useful practices for building and maintaining a computerized item bank. We address storage of item text, graphics, and statistical history. We deal with the creation of paper-and-pencil and computerized tests from an item bank and the use of Item Response Theory (lRT) to calibrate and equate item banks. New directions in item banking …

12. Computerized Testing In Licensure, C. David Vale

Licensure Testing: Purposes, Procedures, and Practices

Computerized testing has come out of the laboratory and into the field. By rough estimates, over a million licensure and certification examinations are currently given by computer each year, and the number is rising. Computerized testing is not appropriate for every application, however. Computerized tests always result in significantly greater direct costs than paper-and-pencil tests. To justify their use, a computerized test must result in a net dollar saving. This means that something in the process of computerization must offer a cost reduction that more than offsets the direct cost of computerization. The purpose of this chapter is to identify …

13. Future Psychometric Practices In Licensure Testing, Steven S. Nettles

Licensure Testing: Purposes, Procedures, and Practices

New technologies continue to emerge each year, and influence testing practices. In particular, in the last 10 years the personal computer has evolved from a curious and minimally useful tool to an indispensable partner in many certification and licensure testing programs. It is involved in every aspect- including candidate scheduling, test assembly, test administration, test scoring and analysis, and score reporting. Initially, it is used to determine the content to be included in the job analysis instrument, and later, to analyze the returned surveys. After the job analysis is completed and test specifications prepared, it can be used to bank …

In The Name Of The Student... What Is Fairness In College Teaching?, Rita Rodabaugh

Professional and Organizational Development Network in Higher Education: Archives

If we remember our own college days, most of us can think of at least one professor who was less than ideal. All of us have had professors who fit one or more of the following descriptions: dull, boring lecturer; confusing and hard to follow; too easy and presents no challenge; and so on. Yet if you describe your worst experience as a student, more than likely it was one in which you were treated unfairly.

For the past two years, much of my research has focused on college students' perceptions of fair practices in the classroom. From this research, …

Emerging Trends In College Teaching For The 21st Century, Milton D. Cox

Professional and Organizational Development Network in Higher Education: Archives

After national calls for the reform of undergraduate education were made a decade ago, students, parents, and legislators began to apply pressure to reestablish the importance of student learning. More recently, central administrators have begun to change reward structures. University-wide community is beginning to be built around teaching. New disciplinary journals that publish the scholarship of teaching are being started, and established ones are gaining respect. National teaching conferences and journals that provide a forum for the scholarship of teaching are expanding. With these emerging opportunities, faculty are going public about their interest in teaching and learning. Over the last …

The Why Of Teacher/Student Relationships, Richard G. Tiberius

Professional and Organizational Development Network in Higher Education: Archives

I heard it again last week, "Yeah, she has a good relationship with her students, but can she teach?", as if the two were completely separate entities, like the icing and the cake. But a teacher's relationship with learners is not icing. It is an essential component of the teaching and learning process. Plumbers and computer technicians may be able to perform useful services on sinks and computers without entering into relation with them or with their owners but teaching simply cannot happen without teachers entering into relation with their students. Moreover, the teacher's success in facilitating learning is directly …

1. Curriculum-Based Measurement, Stanley L. Deno

Curriculum-Based Measurement

In the history of Greek mythology there is a character named Sisyphus who, for sins committed during his lifetime, is condemned to spend eternity pushing a boulder up a hill. No matter how hard Sisyphus tries as he nears the top of the hill, the boulder rolls back down. Sisyphus cannot escape from this continued cycle of effort and failure. Sometimes, when I think about the experiences of many children attempting to learn basic skills in the public schools, I think of the myth of Sisyphus. Too often, it seems to me, no matter how hard they try, they do …

6. Cba: An Assessment Of Its Current Status And Prognosis For Its Future, Mark R. Shinn, Roland H. Good Iii

Curriculum-Based Measurement

The very fact that curriculum-based assessment (CBA) forms the basis of a topic-driven conference at the center of American educational and psychological measurement (i.e., the Buros Institute) is testimony that the strategies are receiving a substantial amount of professional attention. Although debate continues regarding to whom and when the term curriculum-based assessment should be ascribed (Coulter, 1988), without question, its prominence has grown considerably in the last 10 years. Within the last 5 years, school psychology and special education have seen their flagship journals, School Psychology Review and Exceptional Children, devote special volumes to CBA. National organizations such as …

2. A Review Of Curriculum-Based Procedures On Nine Assessment Components, Gerald Tindal

Curriculum-Based Measurement

The purpose of this chapter is to describe curriculum-based procedures from a broad perspective that encompasses the major models appearing in the professional literature in the past 10 years. Rather than simply review the major perspectives, operating assumptions, and implementation directives of these models, however, nine criteria are presented for a uniform comparison. These criteria were implicit in the adoption of curriculum-based measurement (CBM) in Pine County, Minnesota during the initial training and field-based research conducted in the early 1980s. Therefore, they can be used both to structure the review and to provide district personnel a focused evaluation strategy for …

Curriculum-Based Measurement- Complete Work, Jane Close Conoley, Jack J. Kramer

Curriculum-Based Measurement

This volume in the Buros-Nebraska Series on Measurement and Testing provides current information on the development and implementation of curriculum-based measurement. As the title of the volume suggests, effective measurement of children's classroom achievement is not a new problem. Curriculum-based measurement provides an interesting and useful alternative to traditional strategies for assessing academic performance. This volume continues the tradition of including papers given at the annual Buros-Nebraska Symposium on Testing and Measurement as well as additional contributions selected especially for this book. Each of our authors has made significant contributions to the research that has been produced in the area …

4. Academic Skill Assessment: An Evaluation Of The Role And Function Of Curriculum-Based Measurements, Francis E. Lentz, Jack J. Kramer

Curriculum-Based Measurement

In the most meaningful use of the term assessment, important decisions are made daily by teachers based on their assessment of information obtained from student responses to curriculum-related materials. These assessment decisions may include deciding on extra work or deciding to refer a child for learning or behavior problems. The term curriculum-based assessment (CBA) has been used to encompass a wide range of procedures ranging from these daily informal analyses by teachers, to highly structured measurement systems used in special education systems. Although well-constructed guides exist for some sets of curriculum-based decisions (e.g., Shinn, 1989), there is inadequate empirical research …

Good Teaching: A Matter Of Living The Mystery, Parker J. Palmer

Professional and Organizational Development Network in Higher Education: Archives

Good teaching cannot be equated with technique. It comes from the integrity of the teacher, from his or her relation to the subject or the student, from the capricious chemistry of it all. The author muses about how to give voice to the questions the students won’t ask but which probably lie on the cutting edge of their learning.

8. Issues In Intelligent Computer-Assisted Instruction: Eval Uation And Measurement, Harold F. O'Neil Jr., Eva L. Baker

The Computer and the Decision-Making Process

In this chapter we plan to explore two issues in the field of intelligent computer assisted instruction (ICAI) that we feel offer opportunities to advance the state of the art. These issues are evaluation of ICAI systems and the use of the underlying technology in ICAI systems to develop tests. For each issue we will provide a theoretical context, discuss key constructs, provide a brief window to the appropriate literature, suggest methodological solutions and conclude with a concrete example of the feasibility of the solution from our own research.

INTELLIGENT COMPUTER-ASSISTED INSTRUCTION (ICAI)

ICAI is the application of artificial intelligence …

5. Use Of Computer Technology In Behavioral Assessments, Thomas R. Kratochwill, Elizabeth J. Doll, W. Patrick Dickson

The Computer and the Decision-Making Process

Major developments in the behavioral assessment field have occurred over the past decade (e.g., Barlow, 1981 ; Ciminero, Calhoun, & Adams , 1986; Haynes & Wilson, 1979; Mash & Terdal, 1988a). The use of computer technology by behavioral assessors has occurred, but this is a relatively recent development (Kratochwill, Doll, & Dickson, 1986; Romanczyk, 1986). Consider, for example, that behavioral assessment texts include little discussion of computer applications and many articles restrict discussion of behavioral assessment to observational measures (see Cone & Hawkins, 1977, for an exception). In psychology and education, issues of journals have been devoted to computer applications …

6. The Use Of The Computer In The Practice Of Industrial/ Organizational Psychology, Lyle F. Schoenfeldt, Jorge L. Mendoza

The Computer and the Decision-Making Process

The rapid proliferation of computer technology, in the form of mainframe computers, networks of interconnected machines, and stand-alone personal computers, is having a profound effect on many areas of life. As a result of the spread of computer equipment to offices, homes, and educational institutions; the variety of software applications has grown at an unprecedented rate. With this as background, it should be no surprise that computers have assumed an increasing role in professional practice, including applications in providing services in the area of industrial and organizational psychology.

Industrial-organizational psychologists function in a variety of settings, but primarily provide human …

3. Assessment Of Val Id Ity In Computer-Based Test Interpretations, Kevin L. Moreland

The Computer and the Decision-Making Process

The use of computers to interpret psychological tests is a "hot" topic, both within psychology and without. It is hot in the sense of giving rise to an increasing number of books and articles (e.g., Butcher, 1985, 1987; Eyde, 1987; Krug, 1987). It is hot in the sense of giving rise to an ever-increasing number of business enterprises (compare any recent APA Monitor with an issue from 1981). It is hot in the sense of capturing the attention of the news media (e.g., Petterson, 1983). And it is hot in the sense of giving rise to increasing controversy within psychology …

7. Implementation Decisions In Designing Computer-Based Instructional Testing Programs, John V. Noonan, Paul D. Sarvela

The Computer and the Decision-Making Process

From preschool to graduate school, computer-based instruction (CBI) has become an increasingly common event in today's education and training community. The interactive characteristics of CBI and its ability to simulate advanced concepts and operations, such as patient management simulations for medical students (Whiteside & Whiteside, 1987/88) or the maneuvering of a jet airplane (Conkright, 1982), make CBI an attractive new instructional delivery system for educators working in many different fields .

Because of these qualities , the computer has tremendous potential in educational and psychological measurement. For example, Millman & Arter (1984) describe how the computer aids in maintaining test-item …

6. Measuring Performance In Teacher Assessment, Richard J. Stiggins

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

In performance assessment, the examinee is called upon to display behaviors or produce products which an observer evaluates in terms of pre-specified performance standards. This kind of measurement plays a major role in many school evaluation contexts, including both student and teacher evaluation. The observation and evaluation of a teacher's performance in the classroom represents one of many excellent sources of information on teacher capabilities and the effectiveness of instruction. The purpose of this chapter is to review the various ways performance assessment methodology can serve in teacher-assessment contexts.

The review begins with a summary of the range of settings …

8. Limitations Of Using Student-Achievement Data For Career-Ladder Promotions And Merit-Pay Decisions, Ronald A. Berk

Assessment of Teaching: Purposes, Practices, and Implications for the Profession

A study of U.S. school districts conducted 70 years ago reported that 48% of the districts sampled used merit pay (Evendon, 1918). Since then, the quantity as well as quality of teacher-compensation systems has fluctuated markedly (for details, see Cohen & Murnane, 1985; Murnane & Cohen, 1986; Porwoll, 1979). At present, 29 states are implementing large-scale teacher-incentive programs (a.k.a. career ladder, merit pay, pay for performance), funding local plans, piloting testing models, or using state board of education or legislative mandates to develop programs for teachers and administrators (Southern Regional Education Board, 1986) The status of these programs is summarized …

3. Toward A Cognitive Theory For The Measu Rement Of Achievement, Robert Glaser, Alan Lesgold, Susanne Lajoie

The Influence of Cognitive Psychology on Testing

INTRODUCTION

Given the demands for higher levels of learning in our schools and the press for education in the skilled trades, the professions, and the sciences, we must develop more powerful and specific methods for assessing achievement. We need forms of assessment that educators can use to improve educational practice and to diagnose individual progress by monitoring the outcomes of learning and training. Compared to the well-developed technology for aptitude measurement and selection testing, however, the measurement of achievement and diagnosis of learning problems is underdeveloped. This is because the correlational models that support prediction are insufficient for the task …

5. The Assessment Of Cognitive Factors In Academic Abilities, Stephen L. Benton, Kenneth A. Kiewra

The Influence of Cognitive Psychology on Testing

Nearly 30 years ago, Lee Cronbach (1957) distinguished between the two disciplines of correlational psychology, which investigated naturally occurring individual variance in behavior, and experimental psychology, which examined the effectiveness of certain treatments on behavior. Essentially, correlational psychology examined individual differences using factor analytic techniques; whereas experimental psychology attempted to eliminate individual differences using appropriate interventions. Cronbach believed that these two disciplines should join together to promote aptitude-treatment interaction (A Tl) research that would identify effective treatments for certain types of individuals. With this combined approach, different treatments could be prescribed for skilled and less skilled individuals.

The A Tl …

8. Achievement Test Items: Current Issues, Robert L. Ebel

Social and Technical Issues in Testing: Implications for Test Construction and Usage

The writer of achievement test items is confronted with two major problems, as Lindquist pointed out nearly half a century ago (Lindquist, 1936, p. 17). The first of these is the problem of what to measure. The second is how to measure it. The solution proposed for the first problem is to focus primarily on testing for knowledge and only secondarily on testing for abilities. Cognitive abilities, it is reasonable to believe, depend entirely on knowledge. Although the term knowledge, as commonly used, includes both information and understanding, the most useful kind of knowledge, the kind that will occupy our …