Artificial Intelligence and Robotics Commons^™

Is It Getting Better? An Evaluation Of Two Successive Generations Of Chatgpt In Answering Specialized Vascular Surgery Questions, Dongjin Suh, Quang Le, Leana Dogbe, Kedar Lavingia, Michael Amendola

Department Surgery Faculty Publications

Purpose: Large language models (LLMs) can generate clinically relevant text; however, their performance in highly specialized medical domains remains uncertain. This study evaluated ChatGPT-3.5 and ChatGPT-4 (OpenAI) using vascular surgery board–style questions from the Vascular Education and Self-Assessment Program, version 4 (VESAP4) and compared the two public model versions (June and November 2023).

Materials and Methods: All non-image VESAP4 questions (n=384) were presented independently three times to each model version (ChatGPT-3.5 June/November; ChatGPT-4, June/November). Outcomes included accuracy (proportion correct), consistency (same option letter across all three attempts and “consistently correct”), explanation length (word count), and modes of failure classified for …

Uso De Herramientas De Inteligencia Artificial Generativa Por Parte De Docentes En Una Escuela Del Suroeste De Puerto Rico, Glorimar N. Rodríguez Guiliani

Theses and Dissertations

Esta disertación aplicada fue diseñada para investigar el nivel de conocimiento, uso y dificultades que enfrentan los docentes de quinto a duodécimo grado en una escuela privada del suroeste de Puerto Rico respecto a tecnologías emergentes las cuales presentan desafíos significativos para los docentes y los estudiantes. Se exploró la utilización de herramientas de inteligencia artificial generativa (GenAI) como ChatGPT dentro y fuera del aula para actividades pedagógicas y administrativas.

Los hallazgos revelaron una notable carencia en el conocimiento docente sobre el uso y habilidades de la inteligencia artificial. Se identificó, también, una deficiencia en la capacidad de los docentes …

The Reliability Gap: How Traditional Search Engines Outperform Artificial Intelligence (Ai) Chatbots In Rosacea Public Health Information Quality, Houston C. Nelson, Morgan T. Beauchamp, April A. Pace

Department of Medicine Faculty Publications

Background: The internet has become a primary source of health information for the public, with important implications for patient decision-making and public health outcomes. However, the quality and readability of this content vary widely. With the rise of generative artificial intelligence (AI) tools such as ChatGPT and Gemini, new challenges and opportunities have emerged in how patients access and interpret medical information.

Objective: To evaluate and compare the quality, credibility, and readability of consumer health information provided by traditional search engines (Google, Bing) and generative AI platforms (ChatGPT, Gemini) using three validated instruments: DISCERN, JAMA Benchmark Criteria, and Flesch-Kincaid Readability …

Analysing Nontraditional Students' Chatgpt Interaction, Engagement, Self-Efficacy And Performance: A Mixed-Methods Approach, Mohan Yang, Shiyan Jiang, Belle Li, Kristin Herman, Tian Luo, Shanan Chappell Moots, Nolan Lovett

STEMPS Faculty Publications

Generative artificial intelligence brings opportunities and unique challenges to nontraditional higher education students, stemming, in part, from the experience of the digital divide. Providing access and practice is critical to bridge this divide and equip students with needed digital competencies. This mixed-methods study investigated how nontraditional higher education students interact with ChatGPT in multiple courses and examined relationships between ChatGPT interactions, engagement, self-efficacy and performance. Data were collected from 73 undergraduate and graduate students through chat logs, course reflections and artefacts, surveys and interviews. ChatGPT interactions were analysed using four metrics: prompt number, depth of knowledge (DoK), prompt relevance and …

The Implications Of Chatgpt For Legal Services And Society, Andrew Perlman

Michigan Technology Law Review

On November 30, 2022, OpenAI released a chatbot called ChatGPT.1 To demonstrate the chatbot’s sophistication and its potential implications, both for legal services and society more generally, most of this paper was generated in about an hour through prompts within ChatGPT. Only this abstract, the preface, the outline headers, the footnotes, the epilogue, and the prompts were written by a person. ChatGPT generated the rest of the text with no human editing. To be clear, the responses generated by ChatGPT were imperfect and at times problematic, and the use of an AI tool for law-related services raises a host of …

The Promise And Perils Of Smart (City) Bots As Educational Tools, Thomas Menkhoff, Siew Ning Kan, Shaohui Foong

Research Collection Lee Kong Chian School Of Business

Written from an educator’s perspective, the paper aims to examine the increasing use of chatbot technology capable of answering user queries in various fields such as commerce, public service delivery, urban areas, higher education etc. by simulating human conversation through text messages or voice commands. We analyse why smart bots have become popular technologies in the context smart cities; shed light on different types of bots (chatbots, large language models such as ChatGPT, flying bots); and explain how we use bots as teaching tool in an introductory AI for business course. A key argument putting forward is that smart bots …

Assessing The Accuracy And Utility Of Chatgpt Responses To Patient Questions Regarding Posterior Lumbar Decompression, Alec Giakas, Rajkishen Narayanan, Teeto Ezeonu, Jonathan Dalton, Yunsoo Lee, Tyler Henry, John Mangan, Gregory Schroeder, Alex Vaccaro, Christopher Kepler

Department of Orthopaedic Surgery Faculty Papers

Aim: To examine the clinical accuracy and applicability of ChatGPT answers to commonly asked questions from patients considering posterior lumbar decompression (PLD). Methods: A literature review was conducted to identify 10 questions that encompass some of the most common questions and concerns patients may have regarding lumbar decompression surgery. The selected questions were then posed to ChatGPT. Initial responses were then recorded, and no follow-up or clarifying questions were permitted. Two attending fellowship-trained spine surgeons then graded each response from the chatbot using a modified Global Quality Scale to evaluate ChatGPT’s accuracy and utility. The surgeons then analyzed each question, …

Artificial Intelligence In Orthopaedic Education: A Comparative Analysis Of Chatgpt And Bing Ai’S Orthopaedic In-Training Examination Performance, Clark Chen, Vivek Biololikar, Duncan Vannest, James Raphael, Gene Shaffer

Einstein Health Papers

Background: This study evaluated the performance of generative artificial intelligence (AI) models on the Orthopaedic In-Training Examination (OITE), an annual exam administered to U.S. orthopaedic residency programs. Methods: ChatGPT 3.5 and Bing AI GPT 4.0 were evaluated on standardised sets of multiple-choice questions drawn from the American Academy of Orthopaedic Surgeons OITE online question bank spanning 5 years (2018–2022). A total of 1165 questions were posed to each AI system. The performance of both systems was standardised using the latest versions of ChatGPT 3.5 and Bing AI GPT 4.0. Historical data of resident scores taken from the annual OITE technical …

Cognitive Technologies, Tom Davenport

Asian Management Insights

AI and the revolution of work.

Professor Tom Davenport, the President’s Distinguished Professor of Information Technology and Management at Babson College, speaks about how companies can integrate generative Artificial Intelligence (GenAI) into their operations while ensuring workforce adaptation and skills development.

Do You Say Please Or Thank You To Chatgpt? The Subtle Influence Of Prompt Engineering On Digital Civility, Essraa Nawar

Library Articles and Research

"

In the dynamic and rapidly evolving landscape of artificial intelligence, tools like ChatGPT have become integral to our daily digital interactions. These advanced AI systems assist us with a myriad of tasks, from generating text and answering complex questions to providing creative solutions. However, as we engage more frequently with these non-human entities, an intriguing question arises: Are we inadvertently becoming ruder in our digital communications, or are we consciously maintaining our ingrained habits of politeness?"

Does Generative Ai Facilitate Investor Trading? Evidence From Chatgpt Outages, Qiang Cheng, Pengkai Lin, Yue Zhao

Research Collection School Of Accountancy

In this paper, we use ChatGPT outages to investigate whether investors rely on generative artificial intelligence (GAI) to perform trading-related tasks and the associated impact on stock price informativeness. We first document a significant decline in stock trading volume during ChatGPT outages and find that the effect is stronger for firms with corporate news released immediately before or during the outages. We further document similar declines in the short-run price impact, return variance, and bid-ask spreads, consistent with a reduction in informed trading during the outage periods. Lastly, we use trading volume changes during outages to construct a firm-level measure …

Refining Chatgpt-Generated Code: Characterizing And Mitigating Code Quality Issues, Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Kla Tantihamthavorn, Li Li, Xuan-Bach Dinh Le, David Lo

Research Collection School Of Computing and Information Systems

Since its introduction in November 2022, ChatGPT has rapidly gained popularity due to its remarkable ability in language understanding and human-like responses. ChatGPT, based on GPT-3.5 architecture, has shown great promise for revolutionizing various research fields, including code generation. However, the reliability and quality of code generated by ChatGPT remain unexplored, raising concerns about potential risks associated with the widespread use of ChatGPT-driven code generation.In this article, we systematically study the quality of 4,066 ChatGPT-generated programs of code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is threefold. First, …

Ai: Is The Future Robots?, Ruthanne Fischer

University Honors College

The goal of this thesis is to analyze how Artificial Intelligence is actively changing. The newly developed technology that is known as Artificial Intelligence can potentially have implications that are not yet known. Artificial Intelligence impacts the business world in many ways. Things such as ChatGPT and Deepfakes are some of the biggest forms of Artificial Intelligence today. It is important to know how Artificial Intelligence is changing every day because no one knows the true implications of a technology like this. It is crucial for companies to understand how this technology can potentially be harmful to their employees. Artificial …

Evaluating Introductory Computer Science Labs In The Presence Of Ai Tools, Nicholas Snow, Devin Chaimberlain, Abigail Pitcairn, Benjamin Sweeney

Thinking Matters Symposium

This study explores the resistance of introductory computer science lab assignments to “shortcutting” by generative AI tools, such as ChatGPT. By analyzing the work of three distinct student personas on these assignments, we identified key characteristics of language and structure that influence an assignment's vulnerability to AI abuse. Based on these insights, we propose strategies for educators to adapt labs to both counteract AI shortcutting and encourage productive uses of AI.

Artificial Sociality, Simone Natale, Iliana Depounti

Human-Machine Communication

This article proposes the notion of Artificial Sociality to describe communicative AI technologies that create the impression of social behavior. Existing tools that activate Artificial Sociality include, among others, Large Language Models (LLMs) such as ChatGPT, voice assistants, virtual influencers, socialbots and companion chatbots such as Replika. The article highlights three key issues that are likely to shape present and future debates about these technologies, as well as design practices and regulation efforts: the modelling of human sociality that foregrounds it, the problem of deception and the issue of control from the part of the users. Ethical, social and cultural …

Experience Report: Identifying Common Misconceptions And Errors Of Novice Programmers With Chatgpt, Hua Leong Fwa

Research Collection School Of Computing and Information Systems

Identifying the misconceptions of novice programmers is pertinent for informing instructors of the challenges faced by their students in learning computer programming. In the current literature, custom tools, test scripts were developed and, in most cases, manual effort to go through the individual codes were required to identify and categorize the errors latent within the students' code submissions. This entails investment of substantial effort and time from the instructors. In this study, we thus propose the use of ChatGPT in identifying and categorizing the errors. Using prompts that were seeded only with the student's code and the model code solution …

Exploring The Potential Of Chatgpt In Automated Code Refinement: An Empirical Study, Qi Guo, Shangqing Liu, Junming Cao, Xiaohong Li, Xin Peng, Xiaofei Xie, Bihuan Chen

Research Collection School Of Computing and Information Systems

Code review is an essential activity for ensuring the quality and maintainability of software projects. However, it is a time-consuming and often error-prone task that can significantly impact the development process. Recently, ChatGPT, a cutting-edge language model, has demonstrated impressive performance in various natural language processing tasks, suggesting its potential to automate code review processes. However, it is still unclear how well ChatGPT performs in code review tasks. To fill this gap, in this paper, we conduct the first empirical study to understand the capabilities of ChatGPT in code review tasks, specifically focusing on automated code refinement based on given …

Using Chatgpt To Generate Gendered Language, Shweta Soundararajan, Manuela Nayantara Jeyaraj, Sarah Jane Delany

Conference papers

Gendered language is the use of words that denote an individual's gender. This can be explicit where the gender is evident in the actual word used, e.g. mother, she, man, but it can also be implicit where social roles or behaviours can signal an individual's gender - for example, expectations that women display communal traits (e.g., affectionate, caring, gentle) and men display agentic traits (e.g., assertive, competitive, decisive). The use of gendered language in NLP systems can perpetuate gender stereotypes and bias. This paper proposes an approach to generating gendered language datasets using ChatGPT which will provide data for data-driven …

Opportunities And Limitations: Integrating Narrative Ai Into Game-Based Assessment Creation And Evaluating Student Impacts, Kevin B. Patterson

Theses and Dissertations

The Department of Defense (DoD) has identified the need for a technically proficient workforce in the areas of science, technology, engineering, and mathematics. To meet this need, the DoD is actively seeking innovative technology capable of creating workforce development opportunities that are both accessible and effective. Educational research indicates serious games provide a potential avenue to achieve this goal. Unfortunately, a limited number of tools that simplify the game development process and leverage artificial intelligence are available. Content-generating artificial intelligence might help reduce instructor and game-based assessment designers' workloads while promoting individualized learning in students. This research presents a novel …

Chatgpt Can Offer Satisfactory Responses To Common Patient Questions Regarding Elbow Ulnar Collateral Ligament Reconstruction, William Johns, Alec Kellish, Dominic Farronato, Michael G. Ciccotti, Sommer Hammoud

Rothman Institute Papers

PURPOSE: To determine whether ChatGPT effectively responds to 10 commonly asked questions concerning ulnar collateral ligament (UCL) reconstruction.

METHODS: A comprehensive list of 90 UCL reconstruction questions was initially created, with a final set of 10 "most commonly asked" questions ultimately selected. Questions were presented to ChatGPT and its response was documented. Responses were evaluated independently by 3 authors using an evidence-based methodology, resulting in a grading system categorized as follows: (1) excellent response not requiring clarification; (2) satisfactory requiring minimal clarification; (3) satisfactory requiring moderate clarification; and (4) unsatisfactory requiring substantial clarification.

RESULTS: Six of 10 ten responses were …

A Comparative Study Of Responses To Retina Questions From Either Experts, Expert-Edited Large Language Models, Or Expert-Edited Large Language Models Alone, Prashant D. Tailor, Lauren A. Dalvin, John J. Chen, Raymond Iezzi, Timothy W. Olsen, Brittni A. Scruggs, Andrew J. Barkmeier, Sophie J. Bakri, Edwin H. Ryan, Peter H. Tang, D. Wilkin Parke, Peter Belin, Jayanth Sridhar, David Xu, Ajay E. Kuriyan, Yoshihiro Yonekawa, Matthew R. Starr

Wills Eye Hospital Papers

OBJECTIVE: To assess the quality, empathy, and safety of expert edited large language model (LLM), human expert created, and LLM responses to common retina patient questions.

DESIGN: Randomized, masked multicenter study.

PARTICIPANTS: Twenty-one common retina patient questions were randomly assigned among 13 retina specialists.

METHODS: Each expert created a response (Expert) and then edited a LLM (ChatGPT-4)-generated response to that question (Expert + artificial intelligence [AI]), timing themselves for both tasks. Five LLMs (ChatGPT-3.5, ChatGPT-4, Claude 2, Bing, and Bard) also generated responses to each question. The original question along with anonymized and randomized Expert + AI, Expert, and LLM …

Does Chatgpt Know Calculus?, Kris H. Green

Journal of Humanistic Mathematics

Academics and educators across the world are grappling with how OpenAI’s new software, ChatGPT, will impact teaching and learning. This essay explores ChatGPT’s response to a typical calculus problem as a way of illustrating its functionality and limitations.

Natural Language Processing Analysis Of Online Reviews For Small Business: Extracting Insight From Small Corpora, Benjamin J. Mccloskey, Phillip M. Lacasse, Bruce A. Cox

Faculty Publications

Receiving and acting on customer input is essential to sustaining and growing any service organization, particularly a small family business whose livelihood depends on strong relationships with its customers. The competitive advantage offered by advanced analytical approaches for supporting decisions is not trivial, and enterprises across virtually all domains of society are investing heavily in this emerging discipline. Natural Language Processing (NLP) is a subset of computer science that employs computational approaches to analyze human language; it is effective at extracting insight from text data but frequently requires large corpora to train its models, in the scale of thousands or …

Usage And Knowledge Of Online Tools And Generative Ai: A Survey Of Students, Rahul R. Divekar Phd, Lisette Gonzalez, Sophia Guerra, Natasha Boos

Department of Experience Design (XD) Faculty Publications

Artificial Intelligence (AI) tools like ChatGPT are poised to transform student and educator workflows in higher education. However, there is less documentation on the range of tools students in higher education use, how they use them and in coordination with other online tools for learning, and their expertise using AI tools. We present a mixed-method analysis of a survey conducted at a doctoral-granting university in the United States investigating the adoption of AI tools in the context of other technologies. The findings include how the students used GenAI tools in light of other on-line technologies, their perception of expertise on …

Ethical Education Data Mining Framework For Analyzing And Evaluating Large Language Model-Based Conversational Intelligent Tutoring Systems For Management And Entrepreneurship Courses, Joseph Benjamin R. Ilagan, Jose Ramon Ilagan, Ma. Mercedes T. Rodrigo

Quantitative Methods and Information Technology Faculty Publications

Educational data mining (EDM) can be used to design better and smarter learning technology by finding and predicting aspects of learners. Amend if necessary. Insights from EDM are based on data collected from educational environments. Among these educational environments are computer-based educational systems (CBES) such as learning management systems (LMS) and conversational intelligent tutoring systems (CITSs). The use of large language models (LLMs) to power a CITS holds promise due to their advanced natural language understanding capabilities. These systems offer opportunities for enriching management and entrepreneurship education. Collecting data from classes experimenting with these new technologies raises some ethical challenges. …

33. Chatgpt Assistance In Creating Chemistry Practice Problems: Pitfalls, Positives, And Possibilities, Michael A. Christiansen

Teaching and Generative AI: Pedagogical Possibilities and Productive Tensions

ChatGPT, a relatively new Large Language Model (LLM) artificial intelligence (Al) chatbot, has gained significant attention as the most downloaded software app, accruing over 100 million users within two months of its release. This software can generate quick, articulate responses to virtually any textual query. Many educators are concerned about its potential for enabling student cheating. However, it currently suffers significant limitations in solving chemistry problems-documented through peer-reviewed articles and from the author's experience suggesting that chemistry students will perform far better, on average, by studying than by relying on ChatGPT. This is particularly true of math-centric problems, which the …

9. Uses, Benefits, And Limitations Of Ai Chatbots: Implementing Chatgpt In The First-Year Writing Classroom, Walker Smith, Cora Alward, Jacob Morris, Lydia Peach, Veronica Pulley, Parker Routt, Kolby Sanders, Josh Vogeler

Teaching and Generative AI: Pedagogical Possibilities and Productive Tensions

This essay shares the findings of a Spring 2023 honors composition course in which students conducted their own primary research project to investigate the uses and limitations of ChatGPT for first-year writers. By experimenting with integrating ChatGPT into various stages of their writing and revision processes, the students identified the strengths and weaknesses of the writing tool. Then, they observed and interviewed faculty as they graded different versions of essay drafts with varying levels of ChatGPT assistance. Ultimately, they found that ChatGPT was incredibly useful for certain small-scale tasks, but posed significant risks, especially for students with less experience in …

26. Working Alongside, Not Against, Ai Writing Tools In The Composition Classroom: A Dialectical Retrospective, Daniel Frank, Jennifer K. Johnson

Teaching and Generative AI: Pedagogical Possibilities and Productive Tensions

This article presents a dialectical retrospective on thoughtfully integrating Generate AI tools such as ChatGPT into composition classrooms. Drawing on their experiences and research, the authors outline key principles for using AI as a supplemental aid rather than a replacement for student writing, promoting academic integrity, and fostering critical perspectives on the technology's capabilities and limitations. They share experimental classroom activities and assignments that engage students in hands-on exploration and reflection on their AI-assisted writing processes. Student responses reveal nuanced engagement with the tools to support rather than shortcut learning. The authors argue that attempting to simply prohibit AI use …

Integrating Generative Artificial Intelligence With Systems Architecting Diagram Creation: Advancement, Challenges, Opportunities And Future Perspectives, Cansu Yalim, Holly H. Handley

Engineering Management & Systems Engineering Faculty Publications

Generative AI (GenAI) serves as a powerful tool that can create a wide range of content, including but not limited to text, speech, images, code, videos, and 3D models. ChatGPT stands out as a particularly appealing Generative Pretrained Transformer (GPT) model that offers supplementary capabilities through GPTs and plugins. These extensions enable users to engage with the chatbot and improve its functionality, surpassing mere content generation. Our study delves into the potential of ChatGPT, specifically GPT-4, to expedite the creation of diagrams to support the system architecting process. To this end, we explored the use of ChatGPT's Diagrams Show Me …

Higher Education Faculty Perceptions Of Chatgpt And The Influencing Factors: A Sentiment Analysis Of X, Yoseph Mamo, Helen Crompton, Diane Burke, Christine E. Nickel

STEMPS Faculty Publications

ChatGPT, an AI chatbot developed by OpenAI, was released in November 2022, sparking a significant surge in global awareness and utilization of generative AI across various domains. Although recent studies have acknowledged the significance of ChatGPT in the education sector, they have yet to focus on exploring faculty attitudes toward ChatGPT. We gathered a comprehensive corpus of tweets containing “#ChatGPT” and “#highered” between November 30th, 2022, and April 30th, 2023. We analyzed data by triangulating VADER, NRC lexicon, and ground coding. Findings suggest that 40% of the expressed sentiments were positive, 51% were neutral, and 9% were negative. The study …