Education Commons^™

Do Effect-Size Measures Measure Up?: A Brief Assessment, Anthony J. Onwuegbuzie, Joel R. Levin, Nancy L. Leech

Nancy Leech

Because of criticisms leveled at statistical hypothesis testing, some researchers have argued that measures of effect size should replace the significance-testing practice. We contend that although effect-size measures have logical appeal, they are also associated with a number of limitations that may result in problematic interpretations of them in research on children and adults with learning disabilities (LD). The purpose of the present paper is to provide a framework for reporting and interpreting empirical research findings in LD research. Specifically, we recommend that: (1) researchers apply criteria of both statistical significance and substantive significance to help consumers of research assess …

Pilot Study For Validity And Reliability Of An Aptitude Test, Bahram Kazemian

Bahram Kazemian

The study was conducted in the department of the English University of Gujrat during Spring- 2012 semester. A question paper was designed to check the aptitude of the intermediate students of population 25. There were three sections; Grammar, vocabulary and reading comprehension, in the question paper. Section: A (Grammar) was proved valid with 84.33 % of validity. The validity of Section: B (vocabulary) and Section C (reading comprehension) were 91.64 % and 52.00 respectively. As a whole, the validity of all the questions was 75.99 %. Thus, the designed aptitude test may be considered reliable.

Factors That Influence The Difficulty Of Problem Solving Items, Dara Ramalingam, Ray Philpot

Ray Philpot

Computer-based assessment of problem solving allows problems of both static and interactive natures to be posed. Examples of static problems are scheduling and logic puzzles in which all relevant information is available to the solver at the outset. Problems of an interactive nature, on the other hand, require exploration of the situation to acquire additional knowledge needed to solve the problem. Examples include discovering how to use an unfamiliar mobile telephone or automatic vending machine. This study used data from the 2011 Field Trial of the PISA 2012 computer-based assessment of problem solving which comprised 34 static and 45 interactive …

Tyranny Of The Meritocracy?: A Disputation Over Testing With Professor Lani Guinier, Dan Subotnik

Dan Subotnik

No abstract provided.

Determining The Quality Of Assessment Items In Collaborations: Aspects To Discuss To Reach Agreement, Lambert Schuwirth, Jacob Pearce

Dr Jacob Pearce

No abstract provided.

Evaluation Of Item Parameter Recovery Estimation By Acer Conquest Software, Luc T. Le, Ray Adams

Dr Luc Tu Le

ACER ConQuest (Adams, Wu, and Wilson, 2012) has been popularly used for analysing testing and assessment data. Two of the most common estimation methods for Rasch measurement models (Rasch, 1960/1980) are available in this software, marginal maximum likelihood estimation (MML) and joint maximum likelihood estimation (JML). This study is concerned with item parameter recovery for the dichotomous Rasch model. Our primary focus is on comparing JML and MML when the assumptions of MML are violated, that is the abilities are not sampled from the distribution that is assumed in the estimation.

Evaluation Of Item Parameter Recovery Estimation By Acer Conquest Software, Luc T. Le, Ray Adams

Professor Ray Adams

ACER ConQuest (Adams, Wu, and Wilson, 2012) has been popularly used for analysing testing and assessment data. Two of the most common estimation methods for Rasch measurement models (Rasch, 1960/1980) are available in this software, marginal maximum likelihood estimation (MML) and joint maximum likelihood estimation (JML). This study is concerned with item parameter recovery for the dichotomous Rasch model. Our primary focus is on comparing JML and MML when the assumptions of MML are violated, that is the abilities are not sampled from the distribution that is assumed in the estimation.

The Estimation Of Polytomous Item Response Models With Many Dimensions, Nikolai Volodin, Ray J. Adams

Professor Ray Adams

Identification conditions and an improved estimation method for a D-dimensional mixed coefficients multinomial logit model are discussed. This model is a generalisation of the Adams and Wilson (1997) random coefficients multinomial logit and it can be used to fit multdimensional forms of a wide range of Rasch measurement models. The computational demands of the numerical integration required in fitting such models have limited previous implementations to three and perhaps four-dimensional problems (Glas, 1992; Adams, Wilson and Wang, 1997). This paper illustrates a Monte Carlo integration method that permits the estimation of models with much higher dimensionality. The example in this …

Trialling Of Test Items; Is The Data A Reliable Predictor Of Final Test Performance?, Ross Hudson

Dr Ross Hudson

Trialling is seen as a necessary first step in producing a reliable valid assessment tool. However, how reliable is the trialling result in terms of predicting population test performance?

Pisa Reading Literacy Framework, Juliette Mendelovits

Juliette Mendelovits

No abstract provided.

Adaptive Testing For Psychological Assessment: How Many Items Are Enough To Run An Adaptive Testing Algorithm?, Michaela Wagner-Menghin, Geoff Masters

Prof Geoff Masters AO

Although the principles of adaptive testing were established in the psychometric literature many years ago (e.g., Weiss, 1977), and practice of adaptive testing is established in educational assessment, it is not yet widespread in psychological assessment. One obstacle to adaptive psychological testing is a lack of clarity about the necessary number of items to run an adaptive algorithm. The study explores the relationship between item bank size, test length and measurement precision. Simulated adaptive test runs (allowing a maximum of 30 items per person) out of an item bank with 10 items per ability level (covering .5 logits, 150 items …

Naplan And My School : Shedding Some Light On A Work In Progress, Geoff Masters

Prof Geoff Masters AO

Debate about NAPLAN and the My School website has generated plenty of heat. Geoff Masters casts some light on what is essentially a work In progress.

Computer Adaptive Testing : A Feasibility Study, Siek Khoo, Geoff Masters, Ray Adams

Prof Geoff Masters AO

The Australian National Office of Overseas Skills Recognition (NOOSR) commissioned ACER to investigate the feasibility of implementing computer adaptive testing (CAT) in NOOSR’s screening examinations for overseas trained professionals. The National Office administers screening examinations in seven professions (Dentistry, Dietetics, Occupational Therapy, Podiatry, Pharmacy, Physiotherapy and Veterinary Science) at venues throughout Australia and in up to fifty centres around the world. Surveys were conducted to gain an overview of the current methods and procedures used by NOOSR in the screening examinations. Issues related to the application of CAT to NOOSR’s screening examinations and the possible improvement of NOOSR’s assessment program …

Assessing Science Learning, Geoff Masters

Prof Geoff Masters AO

A new school assessment resource provides teachers with information about individual students’ achievement and progress in science. Geoff Masters details the development of the Progressive Achievement Test in Science.

"Thinking" In A Deweyan Perspective: The Law School Exam As A Case Study For Thinking In Lawyering, Donald J. Kochan

Donald J. Kochan

As creatures of thought, we are thinking all the time, but that does not necessarily mean that we are thinking well. Answering the law school exam, like solving any problem, requires that the student exercise thinking in an effective and productive manner. This Article provides some guidance in that pursuit. Using John Dewey’s suspended conclusion concept for effective thinking as an organizing theme, this Article presents one basic set of lessons for thinking through issues that arise regarding the approach to a law school exam. This means that the lessons contained here help exercise thought while taking the exam — …

Technological Issues For Computer-Based Assessment, Beno Csapó, John Ainley, Randy Bennett, Thibaud Latour, Nancy Law

Dr John Ainley

This chapter reviews the contribution of new information-communication technologies to the advancement of educational assessment. Improvements can be described in terms of precision in detecting the actual values of the observed variables, efficiency in collecting and processing information, and speed and frequency of feedback given to the participants and stakeholders. The chapter reviews previous research and development in two ways, describing the main tendencies in four continents (Asia, Australia, Europe and the US) as well as summarising research on how technology advances assessment in certain crucial dimensions (assessment of established constructs, extension of assessment domains, assessment of new constructs and …

Which Form Of Assessment In A Chemistry Examination Best Describes Student Understanding?, Ross Hudson

Dr Ross Hudson

No abstract provided.

Does Question Type, Content And Gender Influence Student Understanding As Demonstrated In An Entrance Examination?, Ross Hudson, David Treagust

Dr Ross Hudson

The research inquires into the effectiveness of the two predominant forms of questions that are used on the State University Entrance examination for chemistry. These are multiple-choice questions and short-answer questions. This research examines the style of question but also the content type examined (recall and application questions) along with gender differences. The research involved an analysis of previous State University Examinations as well as class trial testing students of both genders on tests designed by the researcher. Rasch analysis of the class trial data was performed allowing comparison of question type and content performance as well as differential analysis …

A Systematic Approach To Literacy Support For First Year Preservice Teachers: Implications For Practice, Pauline Taylor

Associate Professor Pauline Taylor-Guy

Pisa : Frequently Answered Criticisms, Ray Adams

Prof Ray Adams

Studies such as PISA that attempt to compare outcomes across educational systems are expensive and difficult to implement. Further, the results of such studies are routinely criticized by educational commentators - particularly when the results are not consistent with their preconceived ideas about the relative merits and efficiencies of various educational practices and systems. This chapter discusses what is done to ameliorate the threats to the validity of PISA in five areas often targeted by commentators and reviewers as sources of invalidity in international comparisons. The five areas discussed are: (1) sampling - are the samples of students who undertake …

Patscience : Progressive Achievement Tests In Science, Ross Hudson, Ron Martin, Daniel Urbach, Stavroula Zoumboulis

Dr Ross Hudson

The ACER progressive achievement tests in science are for use in Australian schools to provide information to teachers about the level of achievement attained by their students in the concepts, skills and processes of science.

Best Start 2008 : Kindergarten Literacy Assessment, Department Of Education And Training, Nsw : Data Analysis Report, June 2008, Marion O. Meiers, Siek Toon Khoo

Marion Meiers

Language Proficiency And Testing For Migration Purposes: What Are The Practical Implications?, Sacha Develle

Dr Sacha DeVelle

No abstract provided.

Developing Tests And Questionnaires For A National Assessment Of Educational Achievement, Prue Anderson, George Morgan

Prue Anderson

The authors introduce readers to the activities involved in the development of achievement tests, including developing as assessment framework, writing multiple choice and constructed response type items, pretesting, producing test booklets, and handscoring items. A section on questionnaire construction features designing questionnaires, writing questions, coding responses, and linking questionnaire and test score data. The final section covers the development of a test administration manual, selecting test administrators, and contacting sampled schools. A companion CD contains examples of released items from national and international tests, sample questionnaires, and administrative manuals. [Back cover, ed]

Ameliorating Culturally Based Extreme Response Tendencies To Attitude Items, Maurice Walker

Maurice Walker

No abstract provided.

The Influence Of Equating Methodology On Reported Trends In Pisa, Eveline Gebhardt, Ray Adams

Prof Ray Adams

In 2005 PISA published trend indicators that compared the results of PISA 2000 and PISA 2003. This paper explores the extent to which the outcomes of these trend analyses are sensitive to the choice of test equating methodologies, the choice of regression models and the choice of linking items. To establish trends, PISA equated its 2000 and 2003 tests using a methodology based on Rasch Modelling that involved estimating linear transformations that mapped 2003 Rasch-scaled scores to the previously established PISA 2000 Rasch-scaled scores. This paper compares the outcomes of this approach with an alternative, which involves the joint Rasch …

The Impact Of Differential Investment Of Student Effort On The Outcomes Of International Studies, J Butler, Ray Adams

Prof Ray Adams

International comparative assessments of student achievement, such as Trends in Mathematics and Science (TIMSS) and Programme for International Student Achievement (PISA) are becoming increasingly important in the development of evidence-based education policy. The potentially far-reaching influence of such studies underscores the need for these assessments to be valid and reliable. In education, increasing recognition is being given to motivational factors which impact on student learning. This research considers a possible threat to the validity of such studies by investigating the influence the amount of effort invested by test-takers has on their outcomes. Reassuringly, it is found that the reported expenditure …

Modelling Mathematics Problem Solving Item Responses Using A Multidimensional Irt Model, Margaret Wu, Ray Adams

Prof Ray Adams

This research examined students' responses to mathematics problem- solving tasks and applied a general multidimensional IRT model at the response category level. In doing so, cognitive processes were identified and modelled through item response modelling to extract more information than would be provided using conventional practices in scoring items. More specifically, the study consisted of two parts. The first part involved the development of a mathematics problem-solving framework that was theoretically grounded, drawing upon research in mathematics education and cognitive psychology. The framework was then used as the basis for item development. The second part of the research involved the …

High-Stakes Testing: Can Rapid Assessment Reduce The Pressure?, Stuart S. Yeh

Stuart S Yeh

This article presents findings about the implementation of a system for rapidly assessing student progress in math and reading in grades K–12—a system that potentially could reduce pressure on teachers resulting from high-stakes testing and the implementation of the No Child Left Behind Act. Interviews with 49 teachers and administrators in one Texas school district suggest that the assessments allowed teachers to individualize and target instruction; provide more tutoring; reduce drill and practice; and improve student readiness for, and spend more time on, critical thinking activities, resulting in a more balanced curriculum. Teachers reported that the assessments provided a common …

All Is Happening, With Numeracy Included, Dave Tout

David (Dave) Tout

The Adult Literacy and Lifeskills (ALL) Survey (formerly known as the International Life Skills Survey (ILSS)) is a large-scale, comparative survey that goes beyond previous international literacy studies. In addition to the literacy skills measured in the previous International Adult Literacy Survey (IALS), ALL is designed to identify and measure a broader range of skills in the adult population in each participating country. The skills to be directly measured are: prose and document literacy; numeracy; problem solving/analytical reasoning. In addition the assessment will be accompanied by a comprehensive Background Questionnaire, which will collect participant information and indirectly measure two other …