Educational Technology Commons^™

Impact Of Retrieval Augmented Generation And Large Language Model Complexity On Undergraduate Exams Created And Taken By Ai Agents, Erick S. Tyndall, Colleen Gayheart, Alexandre Some, Joseph Genz, Torrey J. Wagner, Brent T. Langhals

Faculty Publications

The capabilities of large language models (LLMs) have advanced to the point where entire textbooks can be queried using retrieval-augmented generation (RAG), enabling AI to integrate external, up-to-date information into its responses. This study evaluates the ability of two OpenAI models, GPT-3.5 Turbo and GPT-4 Turbo, to create and answer exam questions based on an undergraduate textbook. 14 exams were created with four true-false, four multiple-choice, and two short-answer questions derived from an open-source Pacific Studies textbook. Model performance was evaluated with and without access to the source material using text-similarity metrics such as ROUGE-1, cosine similarity, and word embeddings. …

Data Package Supporting Research On Personalized Learning Path Problem Derived From The Cognitive Theory Of Multimedia Learning, Sean A. Mochocki, Mark Reith

Faculty Publications

Personalized Learning Paths (PLP)s are sequences of learning materials and activities that are designed to deliver personalized education to students. Unfortunately, PLPs are often defined and implemented based on faulty cognitive science practices. The PLP research community may benefit from a problem domain and data set that is derived from a scientifically supported cognitive science discipline. This data is published to support the paper, “A Two-Stage Multi-Objective Personalized Learning Path Problem Based on the Cognitive Theory of Multimedia Learning.” This data set includes the data used in this paper, the questionnaire used to gather learner profile data, and the python …

Artificial Intelligence In Educational Research And Scholarship: Seven Framings, Eamon Costello, Giselle Ferreira, Stefan Hrastinski, Jason K. Mcdonald, Ahmed Tlili, George Veletsianos, Victoria I. Marin, Henk Huijser, Sharon Altena

Faculty Publications

In a recent interview (Bender et al., 2025), Professor Emily M. Bender discussed the limitations of technical solutions in addressing harmful Artificial Intelligence (AI) bias. She described a particular point we may reach at which technical solutions stop working, and when we should then widen the lens to ask about the problem framing itself. This is a crucial step in any inquiry that is of concern to both novice and experienced researchers alike: moving from problem-solving to problematisation. This commentary aims to provide educational researchers with a glimpse into the wide array of research problems and problematisation of AI in …

Encouraging Self-Regulation In An Online University Course For Preservice Teachers: An Intervention Study, Karen T. Arnesen, Charles R. Graham, Heather Leary, Alison I. Mcmurry

Faculty Publications

Although self-regulation (SR) has long been seen as an important educational concept, we know little about how to foster it in higher education online student populations. This research addresses this need by introducing an SR intervention into a preservice teacher course. The 30 students who participated used an intervention consisting of weekly student goal setting and self-reflection. Using data collected from a self-regulation survey, weekly reflections, final exam reflections, and interviews, all but one student indicated that the process of setting a goal weekly and especially reflecting on their actions during the week increased their self-regulation abilities. More than half …

Behind The Screen: Exploring Parental Roles In K-12 Online Education, Barbara T. Sandberg, Charles R. Graham, Jered Borup, Richard E. West, Rocio Quiroga Velasquez

Faculty Publications

Independent engagement can be challenging for K-12 online students. Parents’ physical presence makes them an important source of engagement support. Previous research on parental support in online courses is limited and focused on asynchronous learning within a single program. Since the COVID-19 pandemic, online enrollments have grown and instructional modalities have evolved. To better understand parental support post-Covid, we interviewed 21 parents across eight online programs with asynchronous, synchronous, and bichronous course offerings. Parents identified their affective, behavioral, cognitive, and other support roles. Implications for parental roles in addressing the academic engagement gap are provided to both researchers and practitioners.

Theory And Tools: The Future Of Technology-Based Learning (Script), Andrew S. Gibbons Iii

Faculty Publications

We are entering a narrow passage that will see great instability in learning technology, instruction, and distance education. My purpose is to talk about two changes that I believe will redirect the entire future of technology-based education.

Theory And Tools: The Future Of Technology-Based Learning (Slides), Andrew S. Gibbons Iii

Faculty Publications

We are entering a narrow passage that will see great instability in learning technology, instruction, and distance education. My purpose is to talk about two changes that I believe will redirect the entire future of technology-based education.

Data Supporting Research On Personalized Learning Paths, Sean Mochocki, Mark Reith

Faculty Publications

Personalized Learning Paths (PLPs) are a key application of Artificial Intelligence in E-Learning. In contrast to regular Learning Paths, they return a unique sequence of learning materials identified as meeting the individual needs of the students. In the literature, PLPs are often created from knowledge graphs, which assist with ordering topics and their associated learning materials. Knowledge graphs are typically directed and acyclic, to capture prerequisite relationships between topics, though they can also have bidirectional edges when these prerequisite relationships are not necessary. This data package provides a primarily un-directed knowledge graph, with associated repository of open-source learning materials that …

Examining The Self-Regulation Abilities Of Students Enrolled In An Online Secondary Charter School, Karen T. Arnesen, Charles R. Graham, Heather Leary

Faculty Publications

This exploratory research addresses the state of student selfregulation (SR) in an online secondary school. Students are more likely to be successful, especially in online schools, when they are self-regulated. Understanding these students’ current SR abilities can facilitate targeted interventions. Data for this study was gathered from a student survey and interviews of students, parents, and teachers from an online secondary school. Quantitative and qualitative analysis revealed that many students perceived their SR as being adequate to strong and that students were more aware of the help-seeking and time management dimensions of SR than of the other dimensions. This awareness …

Learning From Secondary Blended Teaching Practitioners: Selecting Activities And Connecting In-Person And Online Learning, Michelle Jensen, Charles R. Graham, Cecil R. Short

Faculty Publications

Implementation of blended teaching (BT) practices is increasing rapidly in K-12 classrooms. This trend creates an urgent need for professional development to support BT. An understanding of how experienced teachers implement BT can inform teacher educators and professional development leaders as they develop training to support this practice. We interviewed 24 secondary educators implementing BT strategies to determine what activities they include in the online space and how they connect them to in-person learning. Qualitative thematic analysis of the interviews revealed that experienced blended teachers used a variety of online activities, providing opportunities for learners to interact with content, other …

Parental Support Challenges For K-12 Student Online Engagement, Qi Guo, Charles R. Graham, Jered Borup, Bobbie Sandberg, Richard E. West

Faculty Publications

This research seeks to understand parental support for online students. Using a survey, this study examined perceptions from 567 parents from eight online K-12 programs in a Midwestern state regarding their support for students’ online affective, behavioral, and cognitive engagement. Guided by the Academic Communities of Engagement Framework, our analysis identified eight main challenges parents faced when supporting their online student: (a) focus; (b) keeping on pace; (c) lack of motivation; (d) lack of organization, learning materials, live sessions, and support from online schools; (e) lack of feedback, support, and communication from teachers; (f) parents in need of additional skills …

Speculative Practicescapes Of Learning Design And Dreaming, Eamon Costello, Jason K. Mcdonald, F. Macgilchrist, P. Jandrić, H. Carbonel, S. Crighton, A. Buch, M. A. Peters

Faculty Publications

This article addresses a serious issue that besets learning design: its over-reliance on frameworks that promise particular outcomes for individual learners that accord with pre-defined metrics. This is partly a function of the nature of learning design and devel- opment itself which is commonly seen as outcome-oriented activity that should ben- efit individual learners in specific ways. An alternative approach is adopted here which calls attention to other happenings at the heart of education, including positive emo- tions we experience that are made known through less measurable and more fleeting points of reference. Hence, we draw on sources such as …

The Naci Way: Connecting Native Groups And Teachers Through Culturally Responsive Instructional Design, Stephanie West, Heather Francis, Cally Flox, Brenda Beyal, Emily Soderborg, Jason K. Mcdonald

Faculty Publications

In 2018, the BYU ARTS Partnership Native American Curriculum Initiative (NACI) was developed in response to teacher questions regarding the teaching of Native topics. Despite increased movements towards reconciliation, Native groups continue to be marginalized in Westernized educational settings. Additionally, teachers lack clear guidelines regarding the respectful teaching of Native topics. Describing the challenges we, the NACI team members, faced in our six-year journey partnering with Native groups in Utah, we outline key instructional design decisions we made and identify the culturally responsive principles that guided those decisions. We also advocate for the application of culturally responsive principles and practices …

Academic Communities Of Engagement: Exploring The Impact Of Online And In‑Person Support Communities On The Academic Engagement Of Online Learners, Kristian J. Spring, Charles R. Graham, Courtney N. Hanny, Sara Tuiloma, Kamal Badar

Faculty Publications

Institutions of higher education are interested in supporting the academic success of their students in the most effective ways possible. Increasingly, this means engaging students in a blend of both online and in-person learning activities as well as connecting students with supportive communities in both modalities. This research explores the efforts of an institution that is trying to create positive post-secondary learning experiences for thousands of adults who have experienced significant barriers to accessing higher education through traditional routes. Students around the world are supported through both online and local, in-person communities created to support their academic engagement. In this …

What I Think I Learned, Andrew S. Gibbons Iii

Faculty Publications

A retrospective review of fifty years of research and development experience showing the connectedness of the author’s theoretical ideas to practical application. An effort to show designers how over the span of a career new ideas begin as work-related insights and discoveries that by problem solving flow together to create a unique personal view of design and designing. Encouragement for individual designers to be willing to experiment with new ideas that may step beyond received practice and to learn from those experiences, even to the extent of testing and adopting new worldviews that may differ from the general view. Encouragement …

Learning Experience Design As An Orienting Guide For Practice: Insights From Designing For Expertise, Jason K. Mcdonald, Tyler J. Westerberg

Faculty Publications

In this paper we consider how learning experience design (LXD) improves designers’ capacities to influence learning. We do this by exploring what LXD offers the design of learning environments that help develop learners’ expertise. We discuss how LXD (a) attunes designers to different learning affordances than are emphasized in traditional ID; (b) challenges the universal applicability of common ID techniques; and (c) expands designers’ views of the outcomes for which they can design. These insights suggest that LXD is useful because it refocuses and reframes designers' work around flexible design approaches that are often deemphasized in traditional ID.

Empowering Student Success: Unlocking The Potential Of Project-Based Steel Design Education, Aly Mousaad Aly

Faculty Publications

In the pursuit of student success, it is essential to acknowledge that a singular teaching style does not universally cater to all students. The educator's crucial role lies in creating an optimal learning environment that fosters students' endeavors to excel. This endeavor transcends mere classroom success or employment prospects, encompassing a broader impact on societal well-being. An experiential learning approach, where students actively engage in practical tasks, emerges as the most effective mode of instruction. Integrating project-based learning activities into the curriculum holds immense potential for enhancing student learning. Additionally, the utilization of analysis software tools like FTool and STAAD …

The Everydayness Of Instructional Design And The Pursuit Of Quality In Online Courses, Jason K. Mcdonald

Faculty Publications

This article reports research into the everydayness of instructional design (meaning designers’ daily routines, run-of-the-mill interactions with colleagues, and other, prosaic forms of social contact), and how everydayness relates to their pursuit of quality in online course design. These issues were investigated through an ethnographic case study, centered on a team of instructional designers at a university in the United States. Designers were observed spending significant amounts of time engaged in practices of course refinement, meaning mundane, workaday tasks like revising, updating, fine-tuning, or fixing the courses to which they were assigned. Refining practices were interrelated with, but also experienced …

Informal Practices Of Localizing Open Educational Resources In Ghana, Emily Durham Bradshaw, Jason K. Mcdonald

Faculty Publications

Research on the use of Open Educational Resources (OER) often notes the potential benefits for users to revise, reuse, and remix OER to localize it for specific learners. However, a gap in the literature exists in terms of research that explores how this localization occurs in practice. This is a significant gap given the current flow of OER from higher-income countries in the Global North to lower-income countries in the Global South (King et al., 2018). This study explores how OER from one area of the world is localized when it is used in a different cultural context.

Findings indicated …

“Are These People Real?”: Designing And Playtesting An Alternative Reality, Educational Simulation, Jason K. Mcdonald, Jonathan Balzotti, Melissa Franklin, Jessica Haws, Jamin Rowan

Faculty Publications

In this design case, we report our design and playtest of a form of alternative reality, educational simulation that we call a playable case study (PCS). One of the features that make our simulations unique is how they are designed to implement a principle called This Is Not a Game, or TINAG, meaning that the affordances we design into the simulation suggest to students that the experience they are having is real, in contrast to the way the artificial nature of the experience is highlighted in many computer games. In this case, we describe some challenges we encountered in designing …

“Someone In Their Corner”: Parental Support In Online Secondary Education, Courtney N. Hanny, Charles R. Graham, Richard E. West, Jered Borup

Faculty Publications

Despite increased interest in K–12 online education, student engagement deficits and the resulting student attrition remain widespread issues. The Academic Communities of Engagement (ACE) framework theorizes that two groups support online student engagement: the personal community of support and the course community of support. However, more evidence is needed to understand how members of these communities, especially parents, support students in various contexts. Using insights gleaned from 14 semi-structured interviews of parents with students enrolled in online secondary school, this study adds support to the roles identified in the ACE framework by presenting real examples of parents supporting their online …

Barriers And Enablers To K-12 Blended Teaching, Courtney N. Hanny, Karen T. Arnesen, Qi Guo, Jordan Hansen, Charles R. Graham

Faculty Publications

The attributes of post-pandemic blended teaching (BT) will more closely resemble pre-Coronavirus conditions than the outbreak’s emergency remote teaching. Thus, barriers and enablers for teachers transitioning to BT after emergency remote teaching will be similar to those of early BT adopters. K-12 research has identified external and internal barriers and enablers to technology integration but has largely overlooked those of BT. This study involved analyzing 62 interviews of K-12 blended teachers to discover factors that enabled BT, stood as barriers to initial and continued blending efforts, and helped teachers overcome barriers. Teacher-perceived barriers included, among others, costs perceived as outweighing …

Institutional Support For Academic Engagement In Online And Blended Learning Environments: Exploring Affective, Behavioral, And Cognitive Dimensions, Charles R. Graham, Jered Borup, Sara Tuiloma, Adriana Martínez Arias, Ross Larsen

Faculty Publications

In light of the disruptions caused by the COVID-19 pandemic, leaders of higher education institutions around the world have been contemplating ways to help their universities engage in a digital transformation that must have student engagement and learning as the foremost considerations. This study reports on the work conducted at a university in Colombia that created an evaluation instrument based on the Academic Communities of Engagement (ACE) framework (Borup et al., 2020) to examine how well the institution was supporting the affective, behavioral, and cognitive (ABC) dimensions of engagement in its online and blended learning course offerings. This survey, the …

Digital Learning Transformation In Higher Education: International Cases Of University Efforts To Evaluate And Improve Blended Teaching Readiness, Charles R. Graham, Ganbat Danaa, Tserenchimed Purevsuren, Adriana Martínez, Cinthia Bittencourt Spricigo, Barbara Maria Camilotti, Tserenkhand Batsukh

Faculty Publications

The global pandemic of 2019 brought heightened awareness to institutions of higher education of the need to engage in a digital transformation that extends beyond university business operations to the pedagogy of the classroom. This paper is a case study that explores three international cases of universities in Colombia, Brazil, and Mongolia that are at different stages along the path of a digital pedagogical transformation. This article tells each story, including (1) what is driving the local need to engage in digital transformation, (2) what the major challenges and barriers are to achieving a transformation, and (3) what efforts are …

Understanding The Role Of Online Teaching Assistants In Student Engagement, Sara H. Tuiloma, Charles R. Graham

Faculty Publications

In this paper, we examine how online teaching assistants (OLTAs) interact with students in terms of affective, behavioral, and cognitive support and the influence of OLTA training on these interactions. Through a thematic network analysis, using a tracking instrument and semi-structured interviews with 10 OLTAs, we examined how online university students are supported based on the quantity and quality of interactions with OLTAs, and how training contributes to interactions. Our research addressed two key questions: (1) How do OLTAs support learners in the quantity and quality of interactions? and (2) How does OLTA training around affective and behavioral support influence …

An Exploration Of Institutional And Personal Barriers To Online Academic Engagement At A Brazilian University, Cinthia Bittencourt Spricigo, Bárbara Maria Camilotti, Charles R. Graham, Ruth Baptista

Faculty Publications

The recent global pandemic has raised institutional awareness around the world concerning the importance of having high quality online learning options for students. Learner engagement is often correlated with quality outcomes such as student academic success and student satisfaction. Learner engagement is commonly thought of as having three important dimensions: affective engagement, behavioral engagement, and cognitive engagement (ACE framework). Engagement is also enabled or limited by facilitators/barriers. Three important categories of facilitators/barriers are learner characteristics, personal environment, and course environment. Elements in each of these three areas enable or are barriers to students fully engaging in a course. This research …

Student Retention And Persistence In University Certificate-First Programs, Troy Martin, Randall Davies

Faculty Publications

While access to higher education has grown over the past few years, significant barriers exist for nontraditional students attempting to prepare for and complete postsecondary education. For these students, the traditional methods for acknowledging student achievement do not always work. This research explored the impact of earning professional certificates on nontraditional students’ confidence, motivation, and persistence. Specifically, this study evaluated the matriculation rates between two cohorts of students who participated in the online PathwayConnect program. The mixed-method study found that matriculation rates for students who were encouraged to earn a certificate increased compared to those who followed a traditional path. …

Predicting And Explaining Pre-Service Teachers’ Social Networking Technology Adoption, Nandita Gurjar, Stephen Sivo

Faculty Publications

This research study examined pre-service teachers’ (N=250) intentions to adopt Twitter for professional development. The study used the Technology Acceptance Model to test research hypotheses grounded in the literature. The data were collected with a survey questionnaire and analyzed with Structural Equation Modelling. Findings indicated that ease of use, subjective norms, and perceived connectedness explained the variability in intentions to use Twitter. Perceived mobility, mediated through perceived behavioral control, explained participant differences in the perceived ease of use. Implications for stakeholders include highlighting the role subjective norms and mobile applications play in facilitating the ease of use and connectedness because …

Increase Student Engagement In Online Writing Environments, Jonathan Balzotti, Jason K. Mcdonald, Kevin Haws, Amy Allen Rogers, Matthew J. Baker

Faculty Publications

This case study explores a type of educational simulation, an alternative reality game we call a playable case study (PCS), and how its use influenced student engagement in an online writing classroom. The goal of the simulation was to help students create professional communication artifacts and experience real-world professional communication situations. This article reports the effectiveness of the playable case study as a tool specifically for online writing instruction (OWI). The context of our research was a PCS called Microcore. Acting as interns for a company, students are asked to investigate a serious problem that occurs and present a solution …

Introducing Undergraduates To Instructional Design In A Graduate Studio: An Experiential, Model-Centered Approach, Rebecca Stull Zundell, William Sowards, Scott L. Howell, Jason K. Mcdonald

Faculty Publications

This case study describes a combined graduate and undergraduate instructional design studio that introduced undergraduate students to instructional design in a multifaceted, holistic, and applied way. Reviewing the experience of the undergraduates in the course, this design case describes four learning interventions used to create this applied experience: (1) instructional design team projects—one non-profit and the other in higher education, (2) weekly seminars and biweekly training sessions from field experts, (3) an experiential out-of-state trip, and (4) weekly reflection journals. These studio-based learning interventions are presented within the context of the Experiential Learning Theory and Model-Centered Instruction. Overall, the course …