Educational Assessment, Evaluation, and Research Commons^™

How Co-Designing Computational Modeling Activities Helped Teachers Implement Responsive Teaching Strategies, Hillary Swanson, Luettamae Lawrence, Jared Arnell, Allisia Dawkins, Bonni Jones, Bruce Sherin, Uri Wilensky

Instructional Technology and Learning Sciences Faculty Publications

In recent years, science education has shifted focus, from content to practice. This is reflected in the NGSS, which advocate learning science concepts through engagement in science and engineering practices. Theory building is a central activity of science and computational modeling is a key practice through which contemporary scientists construct theory. In this paper, we discuss an 8th grade science teacher's implementation of a computational modeling lesson. The teacher had co-designed the computational modeling microworld and lesson with the research team over the preceding summers. We investigate the teacher's activity during a whole-class discussion near the end of the lesson, …

Supporting And Sustaining Equitable Steam Activities In High School Classrooms: Understanding Computer Science Teachers’ Needs And Practices When Implementing An E-Textiles Curriculum To Forge Connections Across Communities, Deborah A. Fields, Yasmin B. Kafai

Instructional Technology and Learning Sciences Faculty Publications

While the last two decades have seen an increased interest in STEAM (science, technology, engineering, arts, and mathematics) in K-12 schools, few efforts have focused on the teachers and teaching practices necessary to support these interventions. Even fewer have considered the important work that teachers carry out not just inside classrooms but beyond the classroom walls to sustain such STEAM implementation efforts, from interacting with administrators to recruiting students and persuading parents about the importance of arts and computer science. In order to understand teachers’ needs and practices regarding STEAM implementation, in this paper, we focus on eight experienced computer …

Teaching Reflection On Collaborative Writing In The Spanish Second And Third-Grade Dli Classroom, Lucía Martín García

All Graduate Plan B and other Reports, Spring 1920 to Spring 2023

This portfolio was written and compiled by the author while completing the Utah State University Masters in Second Language Teaching (MSLT) program. The documents included are: a discussion of the author’s teaching environment and experience, a teaching philosophy statement of the approaches that inform the author’s teaching practice, a required professional development peer observation of another instructor, a main paper that investigates benefits of and strategies for collaborative writing in the L2 classroom, and a statement of personal goals for the future. In the main paper, the author offers a concise literature review and a reflection on collaborative writing strategies …

Is Microethnography An Ethnographic Case Study? And/Or A Mini-Ethnographic Case Study? An Analysis Of The Literature, Rebecca Y. Bayeck

Instructional Technology and Learning Sciences Faculty Publications

Selecting the research approach that addresses the research question is often challenging for novice researchers. However, getting a better understanding of the research approaches available in the field, is likely to help novice researchers identify and choose the research approach that fits their situation. In this paper, we discuss microethnography, ethnographic case study, and mini-ethnography case study in order to show that these approaches may have similarities but are different. The author hopes that this discussion will help researchers get a better understanding of these approaches and dissipate the confusion that may exist.

The Impact Of Usu Extension On Positive Youth Development, David Francis

Outcomes and Impact Quarterly

4-H programs are grounded in the belief that youth learn best by doing. Kids and teens complete hands-on projects in many educational areas including science, health, agriculture, and civic engagement. 4-H allows youth to engage in their educational interests in a positive environment where they receive guidance from adult mentors, while also being encouraged to take on proactive leadership roles. Kids can concentrate on one focus area or they can try a variety of programs throughout their 4-H experience. Major educational areas in Utah 4-H at Utah State University (USU) Extension include 4-H STEM, 4-H Healthy Living, and Civic Engagement.

The Impact Of Covid-19 On Students' Academic Performance: The Case Of The University Of Ghana Business School, Joseph Ofori Acheampong

Political Science Student Research

The unruly consequences of the COVID-19 outbreak have hamstrung higher education institutions. This study aims to investigate the perceived impact of COVID-19 on student learning and academic performance through a qualitative approach. In-depth semi-structured interviews with 30 purposively chosen informants were used. The results indicated that 70% of students perceive a paradigm shift in their learning experience before and after COVID-19. It was discovered that their sociodemographic characteristics influenced 60% of students, and 57% were influenced by their parent's socioeconomic status, which influenced their academic achievements during the COVID-19 outbreak. The findings revealed that 63% of students had faced numerous …

Designing Hybrid Human-Ai Orchestration Tools For Individual And Collaborative Activities: A Technology Probe Study, Vanessa Echeverria, Kexin Yang, Luettamae Lawrence, Nikol Rummel, Vincent Aleven

Instructional Technology and Learning Sciences Faculty Publications

Combining individual and collaborative learning is common, but dynamic combinations (which happen as-the-need arises, rather than in pre-planned ways, and may happen on an individual basis) are rare. This work reports findings from a technology probe study exploring alternative designs for classroom co-orchestration support for dynamically transitioning between individual and collaborative learning. The study involved 1) a technology-probe classroom study in an authentic, AI-supported classroom to understand teachers' and students' needs for co-orchestration support over dynamic transitions; and 2) workshops and interviews with students and teachers to get informed feedback about their lived experiences. 118 students and three teachers from …

A Dashboard To Support Teachers During Students’ Self-Paced Ai-Supported Problem-Solving Practice, Vincent Aleven, Jori Blankestijn, Luettamae Lawrence, Tomohiro Nagashima, Niels Taatgen

Instructional Technology and Learning Sciences Faculty Publications

Past research has yielded ample knowledge regarding the design of analytics-based tools for teachers and has found beneficial effects of several tools on teaching and learning. Yet there is relatively little knowledge regarding the design of tools that support teachers when a class of students uses AI-based tutoring software for self-paced learning. To address this challenge, we conducted design-based research with 20 middle school teachers to create a novel real-time dashboard, Tutti, that helps a teacher monitor a class and decide which individual students to help, based on analytics from students’ tutoring software. Tutti is fully implemented and has been …

Positionality: The Interplay Of Space, Context And Identity, Rebecca Y. Bayeck

Instructional Technology and Learning Sciences Faculty Publications

This article considers the way in which positionality shifts and is formed during a cross-cultural study to reveal the complexity of the insider-outsider status. As a researcher in a male-dominated game setting, I reflect on the research process and my interactions with participants to show the interplay of space, context, and identity in shaping a researcher’s status. I discuss the process of gaining access to the research site and participants, and data collection in relation to space, context, and identity. The interaction of my identities with space, and context informed my status at various moments. This interplay constructs a complex …

Co-Designing Ai-Based Orchestration Tools To Support Dynamic Transitions: Design Narratives Through Conjecture Mapping, Luettamae Lawrence, Boyuan Guo, Kexin Yang, Vanessa Echeverria, Zimmy Kang, Vikrant Bathala, Christina Li

Instructional Technology and Learning Sciences Faculty Publications

Dynamically transitioning between individual and collaborative learning has been hypothesized to have positive effects, such as providing the optimal learning mode based on students’ needs. There are, however, challenges in orchestrating these transitions in real-time while managing a classroom of students. AI-based orchestration tools have the potential to alleviate some of the orchestration load for teachers. In this study, we describe a sequence of three design sessions with teachers where we refine prototypes of an orchestration tool to support dynamic transitions. We leverage design narratives and conjecture mapping for the design of our novel orchestration tool. Our contributions include the …

Mapping The Complexities And Benefits Of Research-Design Partnerships, Emma Mercier, Luettamae Lawrence, June Ahn, Christopher Wegemer, Maya Benichou, Yael Kali, Yotam Hod, Marcela Borge, Kimberley Gomez, Ung-Sang Lee, Susan Mckenney, Cindy Poortman, Paula Arce-Trigatti, Britte Haugan Cheng

Instructional Technology and Learning Sciences Faculty Publications

Increasingly our field is recognizing the necessity of close, collaborative relationships with educators, policy makers, students and other potential stakeholders if our design and research work is to have a lasting and more equitable impact on education. However, this work is not easy or quick, and we lack both detailed examples of how it is done and training for new (and current) scholars in how to do it. This symposium brings together a group of scholars who actively engage in RPPs and DBIR in order to highlight the lessons that have been learned and extend our discourse into the realities …

Ai And Ambitious Learning Practices, Cindy E. Hmelo-Silver, Sadhana Puntambekar, Krista Glazewski, Luettamae Lawrence, Nikol Rummel, Vincent Aleven, Gautam Biswas, Suraj Uttamchandani, Asmalina Saleh, Haesol Bae, Thomas Brush, Bradford Mott, James Lester, William Goss, Dana Gnesdilow, Rebecca Passonneau, Purushartha Singh, Chanmin Kim, Marcelo Worsley

Instructional Technology and Learning Sciences Faculty Publications

This symposium will provide opportunities for discussion about how Artificial Intelligence can support ambitious learning practices in CSCL. To the extent that CSCL can be a lever for educational equitable educational change, AI needs to be able to support the kinds of practices that afford agency to students and teachers. However, AI also brings to the fore the need to consider equity and ethics. This interactive session will provide opportunities to discuss these issues in the context of the examples presented here.

Student Centered Language Teaching: A Focus On Student Identity, Rachel Mano

All Graduate Plan B and other Reports, Spring 1920 to Spring 2023

This portfolio is a compilation of essays that describe what the writer has come to see as essential topics in second language acquisition. It begins with a professional environment piece, and then a teaching philosophy statement focused on student identity and interaction in the classroom. This is followed by an essay on observations of teaching. The next two sections focus on pragmatic resistance among advanced learners and the importance of preparing learners for peer interaction. The portfolio concludes with an annotated bibliography outlining the main concepts associated with Communicative Language Teaching, a method that is commonly employed in second language …

Peer-To-Peer Feedback: An Instructional Strategy In Teacher Evaluation, Brady L. Ridge

All Graduate Theses and Dissertations, Spring 1920 to Summer 2023

Over the last several decades there have been federal movements in education which place more accountability on schools and consequently their principals. Principals have many responsibilities which vary from administrative duties to improving teaching in their schools. As a result, principals are often asked to do more than they believe they have time to accomplish. Peer-to-peer feedback is a practice that principals can use to help evaluate, and more importantly, help teachers improve their teachers' teaching. This multi-case or multi-school qualitative study researched the practice of peer-to-peer feedback at two elementary schools. The purpose was to understand how peer-to-peer feedback …

Surveying Teachers' Preferences And Boundaries Regarding Human-Ai Control In Dynamic Pairing Of Students For Collaborative Learning, Kexin Bella Yang, Luettamae Lawrence, Vanessa Echeverría, Boyuan Guo, Nikol Rummel, Vincent Aleven

Instructional Technology and Learning Sciences Faculty Publications

Orchestration tools may support K-12 teachers in facilitating student learning, especially when designed to address classroom stakeholders’ needs. Our previous work revealed a need for human-AI shared control when dynamically pairing students for collaborative learning in the classroom, but offered limited guidance on the role each agent should take. In this study, we designed storyboards for scenarios where teachers, students and AI co-orchestrate dynamic pairing when using AI-based adaptive math software for individual and collaborative learning. We surveyed 54 math teachers on their co-orchestration preferences. We found that teachers would like to share control with the AI to lessen their …

The Intersection Of Cultural Context And Research Encounter: Focus On Interviewing In Qualitative Research, Rebecca Yvonne Bayeck

Instructional Technology and Learning Sciences Faculty Publications

This article discusses the influence of the cultural context on the interview process. With literature demonstrating the role of spatial context on interviews, the article contends that similar consideration should be given to cultural contexts of research studies. Focusing on the cultural context where the interview takes place and the interactions during the interview can help researchers understand and analyze interview material. Interview forms such as conversation/interview bombing emerged from the interaction of cultural context with the interview process. This points to the need for qualitative researchers to explore how the cultural context shapes their research encounter. Such focus will …

Towards Bringing Human-Centered Design To K-12 And Post-Secondary Education, Saadeddine Shehab, Mike Tissenbaum, Luettamae Lawrence, Daniel Rees Lewis, Matthew Easterday, Spencer Carlson, Adam Royalty, Helen Chen, Sheppard Sheri, Shelley Goldman, Annie Camey Kuo, Kimiko Lange, Melissa Mesinas, Rose K. Pozos, Dhvani Toprani, Mona Alqahtani, Yu Xia, Marcela Borge, Keith Sawyer

Instructional Technology and Learning Sciences Faculty Publications

Human-Centered Design (HCD) is a growing field that has the potential to positively impact students’ learning. A general consensus on the terms, practices, scaffolds, and assessments of HCD can foster its effective implementation in K-12 and post-secondary education. This session brings together researchers whose work is focused on implementing HCD across K-16 classrooms. It aims to develop a coherent definition of HCD, its methods, practices, and assessments, to help frame the field and reduce ambiguity at a critical time in its broader adoption.

Examining The Influence Of Instructor Interventions On Group Collaboration, Luettamae Lawrence, Taylor Tucker, Emma Mercier

Instructional Technology and Learning Sciences Faculty Publications

Collaborative problem solving is often used in STEM higher education courses to support conceptual knowledge and teamwork. However, course teaching assistants (TAs) often lack the collaborative pedagogical knowledge necessary to orchestrate this form of learning. In this paper, we examine TAs’ orchestration strategies and technology used to understand how these factors influence groups’ collaboration. Contributions from this paper describe the interplay among technology, strategies, and groups’ collaboration toward understanding how to support collaboration in these courses.

Explorations Of Designing Spatial Classroom Analytics With Virutal Prototyping, Jiwoong Jang, Jaewook Lee, Vanessa Echeverría, Luettamae Lawrence, Vincent Aleven

Instructional Technology and Learning Sciences Faculty Publications

Despite the potential of spatial displays for supporting teachers’ classroom orchestration through real-time classroom analytics, the process to design these displays is a challenging and under-explored topic in the learning analytics (LA) community. This paper proposes a mid-fidelity Virtual Prototyping method (VPM), which involves simulating a classroom environment and candidate designs in virtual space to address these challenges. VPM allows for rapid prototyping of spatial features, requires no specialized hardware, and enables teams to conduct remote evaluation sessions. We report observations and findings from an initial exploration with five potential users through a design process utilizing VPM to validate designs …

Understanding The Transient Nature Of Stem Doctoral Students’ Research Self-Efficacy Across Time: Considering The Role Of Gender, Race, And First-Generation College Status, Kaylee Litson, Jennifer M. Blaney, David F. Feldon

Instructional Technology and Learning Sciences Faculty Publications

Developing research self-efficacy is an important part of doctoral student preparation. Despite the documented importance of research self-efficacy, little is known about the progression of doctoral students’ research self-efficacy over time in general and for students from minoritized groups. This study examined both within- and between-person stability of research self-efficacy from semester to semester over 4 years, focusing on doctoral students in biological sciences (N = 336). Using random intercept autoregressive analyses, we evaluated differences in stability across gender, racially minoritized student status, and first-generation student status. Results showed similar mean levels of self-efficacy across demographic groups and across …

Debugging By Design: A Constructionist Approach To High School Students’ Crafting And Coding Of Electronic Textiles As Failure Artifacts, Deborah A. Fields, Yasmin B. Kafai, Luis Morales-Navarro, Justice T. Walker

Instructional Technology and Learning Sciences Faculty Publications

Much attention in constructionism has focused on designing tools and activities that support learners in designing fully finished and functional applications and artifacts to be shared with others. But helping students learn to debug their applications often takes on a surprisingly more instructionist stance by giving them checklists, teaching them strategies or providing them with test programs. The idea of designing bugs for learning—or debugging by design—makes learners agents of their own learning and, more importantly, of making and solving mistakes. In this paper, we report on our implementation of “Debugging by Design” activities in a high school classroom over …

Leveraging Local Resources And Contexts For Inclusive Computer Science Classrooms: Reflections From Experienced High School Teachers Implementing Electronic Textiles, Mia S. Shaw, Deborah A. Fields, Yasmin B. Kafai

Instructional Technology and Learning Sciences Faculty Publications

Background and context

Promoting open-ended projects presents new opportunities and challenges for inclusive teaching in CS classrooms. While efforts have been made to develop inclusive curricula, little research has focused on ways teachers apply curricula in their classrooms to promote inclusion.

Objective

To understand the challenges faced in facilitating an open-ended unit and the pedagogical strategies enacted to address those challenges, we analyze the self-reported teaching practices that experienced teachers developed in their implementation of a constructionist electronic textiles unit in Exploring Computer Science.

Method

We inductively analyzed and coded 17 experienced teachers’ weekly surveys and post-interviews.

Findings

Teachers …

Developing And Validating Stealth Assessments For An Educational Game To Assess Young Dual Language Immersion Learners' Reading Comprehension, Frederick J. Poole

All Graduate Theses and Dissertations, Spring 1920 to Summer 2023

The purpose of this multiple-paper dissertation is to design a digital game and stealth assessments within the game to assess young second language learners' Chinese reading proficiency. In Chapter 2 (Paper 1), I describe the game designed for this dissertation and how it was implemented in a dual language immersion classroom. This study found that the digital game and in-class implementation led to significant vocabulary and reading comprehension gains. Further, seven types of support that students needed while playing the game were identified. In Chapter 3 (Paper 2), I describe how educational data mining approaches, and more specifically, how data-driven …

Finding A Fit: Biological Science Doctoral Students’ Selection Of A Principal Investigator And Research Laboratory, Michelle A. Maher, Annie M. Wofford, Josipa Roksa, David F. Feldon

Instructional Technology and Learning Sciences Faculty Publications

In the laboratory-based disciplines, selection of a principal investigator (PI) and research laboratory (lab) indelibly shapes doctoral students’ experiences and educational outcomes. Framed by the theoretical concept of person–environment fit from within a socialization model, we use an inductive, qualitative approach to explore how a sample of 42 early-stage doctoral students enrolled in biological sciences programs made decisions about fitting with a PI and within a lab. Results illuminated a complex array of factors that students considered in selecting a PI, including PI relationship, mentoring style, and professional stability. Further, with regard to students’ lab selection, peers and research projects …

Steam Maker Education: Conceal/Reveal Of Personal, Artistic And Computational Dimensions In High School Student Projects, Lindsay Lindberg, Deborah A. Fields, Yasmin B. Kafai

Instructional Technology and Learning Sciences Faculty Publications

Much of maker education’s expansion has focused on STEM (science, technology, engineering, mathematics) learning, leaving out equally promising opportunities for integrating arts learning. In this paper we share findings from a project in which high school students created electronic-textiles-based art representing features of a community important to them as a part of an elective high school computer science class. We addressed the following research questions: (1) What kinds of personal and community meanings did students convey through their maker projects? (2) How did students engage with artistic dimensions in their projects? (3) How did coding interactive features support students’ artwork? …

The Validity And Instructional Value Of A Rubric For Evaluating Online Course Quality: An Empirical Study, Ji Eun Lee, Mimi Recker, Min Yuan

Instructional Technology and Learning Sciences Faculty Publications

This study investigated the validity and instructional value of a rubric developed to evaluate the quality of online courses offered at a midsized public university. This rubric was adapted from an online course quality rubric widely used in higher education, the Quality Matters rubric. We first examined the reliability and preliminary construct validity of the rubric using quality ratings for 202 online courses and eliminated twelve problematic items. We then examined the instructional value of the rubric by investigating causal relationships between 1) course quality scores, 2) online interactions between students, instructors, and content, and 3) student course performance (course …

Consensual Assessment In The New Domain Of E-Textiles: Comparing Insights From Expert, Quasi-Expert, And Novice Judges, Anthony Phonethibsavads, Maggie Dahn, Kylie Peppler, Deborah A. Fields, Yasmin B. Kafai

Instructional Technology and Learning Sciences Faculty Publications

Establishing what constitutes creativity in a domain is something for which we often look to experts—individuals versed in a domain’s history and able to identify timeworn ideas from fresh ones. Such valuations of creative merit are tied to a familiarity with past and present trends and, therefore, opinions of newcomers are often ignored. However, what about domains that build upon new, unexplored practices? This study examines the creativity ratings of judges with varying expertise in the emergent domain of electronic textiles (or e-textiles). E-textiles are fabrics that have programmable electronics such as sensors and actuators embedded in them toward a …

Assessing Community-Engaged Learning Impacts Using Ripple Effects Mapping, Benjamin J. Muhlestein, Roslynn Mccann

Journal on Empowering Teaching Excellence

Communicating Sustainability, an upper level undergraduate service-learning live broadcast course was created at Utah State University to help students gain critical skills in communicating and participating in local sustainability efforts. Community-Engaged Learning was a key component applied in gaining and using these skills. This study sought to capture the impacts of this course on both its students and the community partners who worked with those students using Ripple Effects Mapping. Key findings include: powerful impacts on student learning, growth and ability to engage in local movements; as well as clearly defined benefits for community partners. Included in this study …

Identifying Faculty And Peer Interaction Patterns Of First-Year Biology Doctoral Students: A Latent Class Analysis, Soojeong Jeong, Jennifer M. Blaney, David F. Feldon

Instructional Technology and Learning Sciences Faculty Publications

Faculty and peer interactions play a key role in shaping graduate student socialization. Yet, within the literature on graduate student socialization, researchers have primarily focused on understanding the nature and impact of faculty alone, and much less is known about how peer interactions also contribute to graduate student outcomes. Using a national sample of first-year biology doctoral students, this study reveals distinct categories that classify patterns of faculty and peer interaction. Further, we document inequities such that certain groups (e.g., underrepresented minority students) report constrained types of interactions with faculty and peers. Finally, we connect faculty and peer interaction patterns …

Co-Design Of An Orchestration Tool: Supporting Engineering Teaching Assistants As They Facilitate Collaborative Learning., Luettamae Lawrence, Emma Mercier

Instructional Technology and Learning Sciences Faculty Publications

This paper describes a design-based implementation research (DBIR) project, focused on the co-design and implementation of an orchestration tool for teaching assistants (TAs) in required engineering classes. Building on our collaboration with the engineering department, we identified a need for a tool that provides insight into groups to help TAs intervene in realtime. This paper presents two phases of our iterative co-design process. The first phase includes the initial design of the tool from design workshops with TAs. The second phase focuses on a 16-week implementation of the orchestration tool and reports on interviews with TAs to understand how they …