Educational Assessment, Evaluation, and Research Commons^™

Leveraging Consistency: Evaluating Ai Coders For Qualitative Research, Saríah López-Fierro, Carlos Chiriboga-Calderón, José López-Fierro, Rubén Pacheco-Villamar

Instructional Technology and Learning Sciences Faculty Publications

This study provides insights of using Large Language Models (LLMs) for deductive coding tasks in market research. Deductive coding, which applies predefined codes to text, often demands consistency rather than interpretive nuance. 

We compare five closed-weight LLMs with a group of human coders in tagging instances of “Expansive Framing” across qualitative excerpts, using Fleiss’ Kappa, ANOVA, and Estimated Marginal Means. Results reveal that LLMs exhibit higher consistency and faster processing than humans. These findings suggest that integrating LLMs can improve efficiency, making them an attractive asset for firms managing large datasets.

An End-To-End Concatenated Cnn Attention Model For The Classification Of Lung Cancer With Xai Techniques, Fariha Haque, Mohammad Asif Hasan, Md. Abu Ismali Siddique, Tonmoy Roy, Tonmoy Kanti Shaha, Yamina Islam, Avijit Paul, Muhammad E. H. Chowdhury

Instructional Technology and Learning Sciences Faculty Publications

In the field of medical imaging, deep learning (DL) techniques have made significant contributions to the detection and classification of various cancers. Identifying the precise regions in medical images containing cancerous cells plays a crucial role in the diagnostic process. Early and accurate cancer detection is essential for effective treatment and improved patient outcomes. However, manual diagnosis is labor-intensive, requiring the specialized expertise of radiologists, and the increasing number of cancer cases presents challenges in processing large volumes of image data efficiently. To address these challenges, an end-to-end concatenated Convolutional Neural Network (CNN) attention model has been proposed for automatic …

Integrating Human-Centered Design Into Undergraduate Stem Capstone Courses: A Food Product Development Case Study, Saadeddine Shehab, Dawn Bohn, Lucas O'Bryan, Luettamae Lawrence, Mike Tissenbaum

Instructional Technology and Learning Sciences Faculty Publications

This study examines the integration of human-centered design (HCD) in a Food Science capstone course and explores its impact on students’ learning outcomes and skill development. This case study investigated how 42 undergraduate students engaged with HCD processes—including understanding, synthesizing, ideating, prototyping, and implementing—to develop innovative food products. Through pre- and post-surveys, interaction analysis, and performance assessments, the findings revealed that HCD activities significantly enhanced students’ knowledge, collaboration, creativity, and metacognitive skills. Additionally, students demonstrated improved food science competencies, such as product formulation and sensory evaluation. This study highlights the benefits of embedding HCD within STEM curricula to foster authentic, …

Designing A Human-Centred Learning Analytics Dashboard In-Use, Riordan Alfredo, Vanessa Echeverria, Linxuan Zhao, Luettamae Lawrence, Jie Xiang Fan, Lixiang Yan, Xinyu Li, Zachari Swiecki, Dragan Gašević, Roberto Martinez-Maldonado

Instructional Technology and Learning Sciences Faculty Publications

Despite growing interest in applying human-centred design methods to create learning analytics (LA) systems, most efforts have concentrated on initial design phases, with limited exploration of how LA tools and practices can coevolve during the actual learning and teaching activities. This paper examines how a human-centred LA dashboard can be further refined and adapted by teachers while actively using it in a real-world scenario (i.e., design-in-use), beyond its intended design (i.e., design-for-use). We use instrumental genesis as a theoretical lens to analyze the temporary and permanent instrumentalization of design features and individual and collective instrumentation of the LA …

The Intersection Of Place And Game Design: The Case Of Two Classic African Board Games, Rebecca Y. Bayeck, Joseph M. Bayeck

Instructional Technology and Learning Sciences Faculty Publications

The concept of game design is often associated with a computational creative task or a creation with technology. Automatically linking game design to computational activity or technology may have contributed to the scarce research on the interaction between place and game design. The focus on technology may have excluded indigenous games from game design conversation in the game design literature. This article explores design in the context of two classic African board games. We examine the influence of place on the design of these games. Building on game design and place literature, the article shows the situatedness of game design …

Cognitive Reappraisal: The Bridge Between Cognitive Load And Emotion, Rebecca Brockbank, David F. Feldon

Instructional Technology and Learning Sciences Faculty Publications

Within this integrative review, cognitive load theory (CLT) is asserted as a powerful framework for conceptualizing human cognitive processes within learning. The relationship between cognition and emotion is then examined and further integrated within the scope of CLT. Emotion regulation strategies are discussed and adaptive strategies are proposed as being of particular relevance to broadening the theoretical and practical impacts of CLT. Central to the argument of this review is the use of cognitive reappraisal as a potential mitigator of cognitive load. Cognitive reappraisal involves reframing or reassessing understandings or beliefs that underlie an emotional response, which may mitigate cognitive …

Accuracy And Effectiveness Of An Orchestration Tool On Instructors' Interventions And Groups' Collaboration, Luettamae Lawrence, Emma Mercier, Taylor Tucker Parks, Nigel Bosch, Luc Paquette

Instructional Technology and Learning Sciences Faculty Publications

This paper presents the development of a novel orchestration tool that predicts collaborative problem-solving (CPS) behaviors of undergraduate engineering groups and investigates the use of that tool by instructors. We explore the impact of receiving real-time, machine-learning, model-based prompts on 1) instructors' orchestration strategies, which are strategies instructors use to manage and facilitate collaborative activities, and 2) groups' participation, including how groups are engaged in CPS activities. The orchestration tool is a dashboard that notifies instructors of–and advises them on–monitoring and intervening with groups who may need collaborative support and guidance. We describe the accuracy of the models in predicting …

Computational Models As Tools For Supporting Responsive Teaching, Hillary Swanson, Luettamae Lawrence, Jared Arnell, Bonni Jones, Bruce Sherin, Uri Wilensky

Instructional Technology and Learning Sciences Faculty Publications

It is widely agreed that science instruction should help students build new knowledge on the foundation of their prior knowledge. Responsive teaching refers to a family of teaching strategies that pursue and build on student ideas. We introduce a particular approach to responsive teaching and examine how it can be supported by the use of computational models. We analyze an 8^th grade science teacher's facilitation of a class discussion near the end of a lesson on sound. We present a moment-by-moment characterization of her responsive teaching moves, highlighting the ways she used a computational model to help students articulate …

Meta-Theoretic Competence For Computational Agent-Based Modeling, Hillary Swanson, Uri Wilensky

Instructional Technology and Learning Sciences Faculty Publications

In the U.S. context, science standards encourage educators to engage students in modeling practices, including computational modeling. While much work has investigated the productivity of computational modeling with respect to students' development of scientific content knowledge, less work has focused on students' development of knowledge and skills for participation in computational modeling practices. A first step in understanding how these practices develop is examining students' activity in the context of computational modeling environments with attention to the productive moves they make. These moves can provide insight into the knowledge they bring to their learning, which may be foundational to the …

Design Forms And Games, Hillary Swanson, Ha Nguyen, Caroline Lavoie, Trilby Nelson

Instructional Technology and Learning Sciences Faculty Publications

This paper introduces a theoretical framework for characterizing design processes as design games. Based on Collins and Ferguson's epistemic forms and games, the framework characterizes design processes as games played by designers to fill out design forms, which are general templates underlying specific design artifacts. We introduce our framework through games involved in human-centered design, describing for each the entry conditions, target form, constraints, and moves. We demonstrate how the framework can be used to characterize design games played by experts in museum exhibit design and urban design. Through similar analyses, the framework …

The Role Of Generative Ai In Qualitative Research: Gpt-4'S Contributions To A Grounded Theory Analysis, Ravi Sinha, Idris Solola, Ha Nguyen, Hillary Swanson, Luettamae Lawrence

Instructional Technology and Learning Sciences Faculty Publications

We present reflections on our experience using a generative AI model in qualitative research, to illuminate the AI's contributions to our analytic process. Our analytic focus was a segment of classroom transcript, which captured a teacher introducing scientific theory-building practices to middle school students. We used a grounded theory approach to produce a fine-grained characterization of the teacher's talk moves during the lesson implementation. Our eventual goal is to build a more nuanced conceptualization of responsive teaching in the context of theory-building activities. We involved GPT-4 during the initial exploratory and later focused coding stages. For our analysis of GPT-4's …

What Happened To The Geometry? Examining Spatial And Mathematical Concepts In Computational Toys And Kits For Young Children, Jody Clarke-Midura, Jessica F. Shumway, Anahita Ashineh, Alireza Zandi, Whitney Clawson

Instructional Technology and Learning Sciences Faculty Publications

This paper explores the design of computational toys and kits for young children (ages 4-7) as tools for learning integrated mathematical, spatial, and computational thinking concepts. Specifically, we examine how the design features of the toys and kits represent the concepts of rotation on a point and spatial orientation of the agent. We examine toys and kits sold commercially, developed through research, and used in early childhood classrooms. Our findings indicate that the mathematical and spatial concepts are overlooked in some designs. Prior research examined toys for their affordances related to computational thinking, the present study contributes to understanding of …

Theory-Based Science Communication At Comic Cons, Lisa Lundgren, Emily Slater, Man Zhang, Kadie Kunz, Gabriel-Philip Santos

Instructional Technology and Learning Sciences Faculty Publications

The majority of research concerning science communication happens in spaces where people already have a vested interest in or knowledge of science, such as museums and science cafes. Thus, there is a gap in understanding what theory-based science communication looks like in non-science-centered spaces. This qualitative research study, which featured cosplaying science communicators at comic conventions, offers insights into what science communication practices occur within everyday spaces. We conducted observations of and semi-structured interviews with 15 cosplaying scientists, examining how they utilized effective framing and narrative structuring when communicating science at comic cons in cosplay. Across 700 coded utterances, cosplaying …

Who Gets To Decolonize: A Reflection On The Importance Of Positionality In The Decolonization Of Digital Learning Spaces And Learning Design, Rebecca Y. Bayeck, Tutaleni Iita Asino

Instructional Technology and Learning Sciences Faculty Publications

Decolonization as a term and concept gained prominence in the 20th century as new nations emerged from the cessation of the global colonial project which primarily saw European countries occupying different parts of the world. In modern times, interests in decolonization have taken different forms, particularly in learning design and digital learning spaces. In the digital space, researchers mainly suggest introducing indigenous perspectives and cultural lenses in the design of digital learning environments and the design of emerging technologies that embed indigenous perspectives (Nede, 2023). This reflexive essay discusses the importance of positionality in digital learning spaces and learning/instructional design. …

Understanding Non-Designers' Practices And Processes In A Human-Centered Design Course, Luettamae Lawrence, Saadeddine Shehab, Mike Tissenbaum

Instructional Technology and Learning Sciences Faculty Publications

Human-centered design (HCD) has been identified in the literature as a useful problem-solving approach for learners. However, learning and applying HCD poses several challenges for students who are unfamiliar with this form of learning. In this paper, we analyse how a novice design team worked on a HCD class project to examine how non-designers learn about and integrate HCD practices into their project. We introduce the HCD taxonomy to define the processes and practices that students engage in. The team’s design work is triangulated across multiple data sources and revealed three challenges for engaging non-designers in HCD: 1) the need …

How Teachers Conceptualise Shared Control With An Ai Co-Orchestration Tool: A Multiyear Teacher-Centred Design Process, Luettamae Lawrence, Vanessa Echeverria, Kexin Yang, Vincent Aleven, Nikol Rummel

Instructional Technology and Learning Sciences Faculty Publications

Artificial intelligence (AI) can enhance teachers' capabilities by sharing control over different parts of learning activities. This is especially true for complex learning activities, such as dynamic learning transitions where students move between individual and collaborative learning in un-planned ways, as the need arises. Yet, few initiatives have emerged considering how shared responsibility between teachers and AI can support learning and how teachers' voices might be included to inform design decisions. The goal of our article is twofold. First, we describe a secondary analysis of our co-design process comprising six design methods to understand how teachers conceptualise sharing control with …

How Co-Designing Computational Modeling Activities Helped Teachers Implement Responsive Teaching Strategies, Hillary Swanson, Luettamae Lawrence, Jared Arnell, Allisia Dawkins, Bonni Jones, Bruce Sherin, Uri Wilensky

Instructional Technology and Learning Sciences Faculty Publications

In recent years, science education has shifted focus, from content to practice. This is reflected in the NGSS, which advocate learning science concepts through engagement in science and engineering practices. Theory building is a central activity of science and computational modeling is a key practice through which contemporary scientists construct theory. In this paper, we discuss an 8th grade science teacher's implementation of a computational modeling lesson. The teacher had co-designed the computational modeling microworld and lesson with the research team over the preceding summers. We investigate the teacher's activity during a whole-class discussion near the end of the lesson, …

Supporting And Sustaining Equitable Steam Activities In High School Classrooms: Understanding Computer Science Teachers’ Needs And Practices When Implementing An E-Textiles Curriculum To Forge Connections Across Communities, Deborah A. Fields, Yasmin B. Kafai

Instructional Technology and Learning Sciences Faculty Publications

While the last two decades have seen an increased interest in STEAM (science, technology, engineering, arts, and mathematics) in K-12 schools, few efforts have focused on the teachers and teaching practices necessary to support these interventions. Even fewer have considered the important work that teachers carry out not just inside classrooms but beyond the classroom walls to sustain such STEAM implementation efforts, from interacting with administrators to recruiting students and persuading parents about the importance of arts and computer science. In order to understand teachers’ needs and practices regarding STEAM implementation, in this paper, we focus on eight experienced computer …

Is Microethnography An Ethnographic Case Study? And/Or A Mini-Ethnographic Case Study? An Analysis Of The Literature, Rebecca Y. Bayeck

Instructional Technology and Learning Sciences Faculty Publications

Selecting the research approach that addresses the research question is often challenging for novice researchers. However, getting a better understanding of the research approaches available in the field, is likely to help novice researchers identify and choose the research approach that fits their situation. In this paper, we discuss microethnography, ethnographic case study, and mini-ethnography case study in order to show that these approaches may have similarities but are different. The author hopes that this discussion will help researchers get a better understanding of these approaches and dissipate the confusion that may exist.

Designing Hybrid Human-Ai Orchestration Tools For Individual And Collaborative Activities: A Technology Probe Study, Vanessa Echeverria, Kexin Yang, Luettamae Lawrence, Nikol Rummel, Vincent Aleven

Instructional Technology and Learning Sciences Faculty Publications

Combining individual and collaborative learning is common, but dynamic combinations (which happen as-the-need arises, rather than in pre-planned ways, and may happen on an individual basis) are rare. This work reports findings from a technology probe study exploring alternative designs for classroom co-orchestration support for dynamically transitioning between individual and collaborative learning. The study involved 1) a technology-probe classroom study in an authentic, AI-supported classroom to understand teachers' and students' needs for co-orchestration support over dynamic transitions; and 2) workshops and interviews with students and teachers to get informed feedback about their lived experiences. 118 students and three teachers from …

A Dashboard To Support Teachers During Students’ Self-Paced Ai-Supported Problem-Solving Practice, Vincent Aleven, Jori Blankestijn, Luettamae Lawrence, Tomohiro Nagashima, Niels Taatgen

Instructional Technology and Learning Sciences Faculty Publications

Past research has yielded ample knowledge regarding the design of analytics-based tools for teachers and has found beneficial effects of several tools on teaching and learning. Yet there is relatively little knowledge regarding the design of tools that support teachers when a class of students uses AI-based tutoring software for self-paced learning. To address this challenge, we conducted design-based research with 20 middle school teachers to create a novel real-time dashboard, Tutti, that helps a teacher monitor a class and decide which individual students to help, based on analytics from students’ tutoring software. Tutti is fully implemented and has been …

Positionality: The Interplay Of Space, Context And Identity, Rebecca Y. Bayeck

Instructional Technology and Learning Sciences Faculty Publications

This article considers the way in which positionality shifts and is formed during a cross-cultural study to reveal the complexity of the insider-outsider status. As a researcher in a male-dominated game setting, I reflect on the research process and my interactions with participants to show the interplay of space, context, and identity in shaping a researcher’s status. I discuss the process of gaining access to the research site and participants, and data collection in relation to space, context, and identity. The interaction of my identities with space, and context informed my status at various moments. This interplay constructs a complex …

Co-Designing Ai-Based Orchestration Tools To Support Dynamic Transitions: Design Narratives Through Conjecture Mapping, Luettamae Lawrence, Boyuan Guo, Kexin Yang, Vanessa Echeverria, Zimmy Kang, Vikrant Bathala, Christina Li

Instructional Technology and Learning Sciences Faculty Publications

Dynamically transitioning between individual and collaborative learning has been hypothesized to have positive effects, such as providing the optimal learning mode based on students’ needs. There are, however, challenges in orchestrating these transitions in real-time while managing a classroom of students. AI-based orchestration tools have the potential to alleviate some of the orchestration load for teachers. In this study, we describe a sequence of three design sessions with teachers where we refine prototypes of an orchestration tool to support dynamic transitions. We leverage design narratives and conjecture mapping for the design of our novel orchestration tool. Our contributions include the …

Mapping The Complexities And Benefits Of Research-Design Partnerships, Emma Mercier, Luettamae Lawrence, June Ahn, Christopher Wegemer, Maya Benichou, Yael Kali, Yotam Hod, Marcela Borge, Kimberley Gomez, Ung-Sang Lee, Susan Mckenney, Cindy Poortman, Paula Arce-Trigatti, Britte Haugan Cheng

Instructional Technology and Learning Sciences Faculty Publications

Increasingly our field is recognizing the necessity of close, collaborative relationships with educators, policy makers, students and other potential stakeholders if our design and research work is to have a lasting and more equitable impact on education. However, this work is not easy or quick, and we lack both detailed examples of how it is done and training for new (and current) scholars in how to do it. This symposium brings together a group of scholars who actively engage in RPPs and DBIR in order to highlight the lessons that have been learned and extend our discourse into the realities …

Ai And Ambitious Learning Practices, Cindy E. Hmelo-Silver, Sadhana Puntambekar, Krista Glazewski, Luettamae Lawrence, Nikol Rummel, Vincent Aleven, Gautam Biswas, Suraj Uttamchandani, Asmalina Saleh, Haesol Bae, Thomas Brush, Bradford Mott, James Lester, William Goss, Dana Gnesdilow, Rebecca Passonneau, Purushartha Singh, Chanmin Kim, Marcelo Worsley

Instructional Technology and Learning Sciences Faculty Publications

This symposium will provide opportunities for discussion about how Artificial Intelligence can support ambitious learning practices in CSCL. To the extent that CSCL can be a lever for educational equitable educational change, AI needs to be able to support the kinds of practices that afford agency to students and teachers. However, AI also brings to the fore the need to consider equity and ethics. This interactive session will provide opportunities to discuss these issues in the context of the examples presented here.

Surveying Teachers' Preferences And Boundaries Regarding Human-Ai Control In Dynamic Pairing Of Students For Collaborative Learning, Kexin Bella Yang, Luettamae Lawrence, Vanessa Echeverría, Boyuan Guo, Nikol Rummel, Vincent Aleven

Instructional Technology and Learning Sciences Faculty Publications

Orchestration tools may support K-12 teachers in facilitating student learning, especially when designed to address classroom stakeholders’ needs. Our previous work revealed a need for human-AI shared control when dynamically pairing students for collaborative learning in the classroom, but offered limited guidance on the role each agent should take. In this study, we designed storyboards for scenarios where teachers, students and AI co-orchestrate dynamic pairing when using AI-based adaptive math software for individual and collaborative learning. We surveyed 54 math teachers on their co-orchestration preferences. We found that teachers would like to share control with the AI to lessen their …

The Intersection Of Cultural Context And Research Encounter: Focus On Interviewing In Qualitative Research, Rebecca Yvonne Bayeck

Instructional Technology and Learning Sciences Faculty Publications

This article discusses the influence of the cultural context on the interview process. With literature demonstrating the role of spatial context on interviews, the article contends that similar consideration should be given to cultural contexts of research studies. Focusing on the cultural context where the interview takes place and the interactions during the interview can help researchers understand and analyze interview material. Interview forms such as conversation/interview bombing emerged from the interaction of cultural context with the interview process. This points to the need for qualitative researchers to explore how the cultural context shapes their research encounter. Such focus will …

Towards Bringing Human-Centered Design To K-12 And Post-Secondary Education, Saadeddine Shehab, Mike Tissenbaum, Luettamae Lawrence, Daniel Rees Lewis, Matthew Easterday, Spencer Carlson, Adam Royalty, Helen Chen, Sheppard Sheri, Shelley Goldman, Annie Camey Kuo, Kimiko Lange, Melissa Mesinas, Rose K. Pozos, Dhvani Toprani, Mona Alqahtani, Yu Xia, Marcela Borge, Keith Sawyer

Instructional Technology and Learning Sciences Faculty Publications

Human-Centered Design (HCD) is a growing field that has the potential to positively impact students’ learning. A general consensus on the terms, practices, scaffolds, and assessments of HCD can foster its effective implementation in K-12 and post-secondary education. This session brings together researchers whose work is focused on implementing HCD across K-16 classrooms. It aims to develop a coherent definition of HCD, its methods, practices, and assessments, to help frame the field and reduce ambiguity at a critical time in its broader adoption.

Examining The Influence Of Instructor Interventions On Group Collaboration, Luettamae Lawrence, Taylor Tucker, Emma Mercier

Instructional Technology and Learning Sciences Faculty Publications

Collaborative problem solving is often used in STEM higher education courses to support conceptual knowledge and teamwork. However, course teaching assistants (TAs) often lack the collaborative pedagogical knowledge necessary to orchestrate this form of learning. In this paper, we examine TAs’ orchestration strategies and technology used to understand how these factors influence groups’ collaboration. Contributions from this paper describe the interplay among technology, strategies, and groups’ collaboration toward understanding how to support collaboration in these courses.

Explorations Of Designing Spatial Classroom Analytics With Virutal Prototyping, Jiwoong Jang, Jaewook Lee, Vanessa Echeverría, Luettamae Lawrence, Vincent Aleven

Instructional Technology and Learning Sciences Faculty Publications

Despite the potential of spatial displays for supporting teachers’ classroom orchestration through real-time classroom analytics, the process to design these displays is a challenging and under-explored topic in the learning analytics (LA) community. This paper proposes a mid-fidelity Virtual Prototyping method (VPM), which involves simulating a classroom environment and candidate designs in virtual space to address these challenges. VPM allows for rapid prototyping of spatial features, requires no specialized hardware, and enables teams to conduct remote evaluation sessions. We report observations and findings from an initial exploration with five potential users through a design process utilizing VPM to validate designs …