Open Access. Powered by Scholars. Published by Universities.®
Physical Sciences and Mathematics Commons™
Open Access. Powered by Scholars. Published by Universities.®
- Keyword
-
- Conversational Agent (2)
- Large Language Model (2)
- Recommendation (2)
- Social Network (2)
- Software engineering (2)
-
- Clustering (1)
- Code Generation (1)
- Code Idiom Mining (1)
- Code Model Analysis (1)
- Code Pattern (1)
- Code Search (1)
- Code Suggestion (1)
- Code quality (1)
- Code smells (1)
- Compositional learning (1)
- Configuration Tuning (1)
- Fill-in-the-blank (1)
- LLMs (1)
- Language Model (1)
- Language Models of Code (1)
- Large Language Model (LLM) (1)
- Large language models (1)
- Memorization (1)
- Mistake-based learning (1)
- Multi-Objective Optimization (1)
- Open-Source Software (1)
- Position-guided text prompt (1)
- Qualitative research (1)
- Refactoring (1)
- Software maintenance (1)
Articles 1 - 21 of 21
Full-Text Articles in Physical Sciences and Mathematics
Style: Improving Domain Transferability Of Asking Clarification Questions In Large Language Model Powered Conversational Agents, Yue Chen, Chen Huang, Yang Deng, Wenqiang Lei, Dingnan Jin, Jia Liu, Tat-Seng Chua
Style: Improving Domain Transferability Of Asking Clarification Questions In Large Language Model Powered Conversational Agents, Yue Chen, Chen Huang, Yang Deng, Wenqiang Lei, Dingnan Jin, Jia Liu, Tat-Seng Chua
Research Collection School Of Computing and Information Systems
Equipping a conversational search engine with strategies regarding when to ask clarification questions is becoming increasingly important across various domains. Attributing to the context understanding capability of LLMs and their access to domain-specific sources of knowledge, LLM-based clarification strategies feature rapid transfer to various domains in a posthoc manner. However, they still struggle to deliver promising performance on unseen domains, struggling to achieve effective domain transferability. We take the first step to investigate this issue and existing methods tend to produce one-size-fits-all strategies across diverse domains, limiting their search effectiveness. In response, we introduce a novel method, called STYLE, to …
Chain-Of-Exemplar: Enhancing Distractor Generation For Multimodal Educational Question Generation, Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, Tat-Seng Chua
Chain-Of-Exemplar: Enhancing Distractor Generation For Multimodal Educational Question Generation, Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, Tat-Seng Chua
Research Collection School Of Computing and Information Systems
Multiple-choice questions (MCQs) are important in enhancing concept learning and student engagement for educational purposes. Despite the multimodal nature of educational content, current methods focus mainly on text-based inputs and often neglect the integration of visual information. In this work, we study the problem of multimodal educational question generation, which aims at generating subject-specific educational questions with plausible yet incorrect distractors based on multimodal educational content. To tackle this problem, we introduce a novel framework, named Chain-of-Exemplar (CoE), which utilizes multimodal large language models (MLLMs) with Chain-of-Thought reasoning to improve the generation of challenging distractors. Furthermore, CoE leverages three-stage contextualized …
Clamber: A Benchmark Of Identifying And Clarifying Ambiguous Information Needs In Large Language Models, Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua
Clamber: A Benchmark Of Identifying And Clarifying Ambiguous Information Needs In Large Language Models, Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua
Research Collection School Of Computing and Information Systems
Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct 12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs.Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence …
Self-Chats From Large Language Models Make Small Emotional Support Chatbot Better, Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, Liqiang Nie
Self-Chats From Large Language Models Make Small Emotional Support Chatbot Better, Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, Liqiang Nie
Research Collection School Of Computing and Information Systems
Large Language Models (LLMs) have shown strong generalization abilities to excel in various tasks, including emotion support conversations. However, deploying such LLMs like GPT-3 (175B parameters) is resource-intensive and challenging at scale. In this study, we utilize LLMs as “Counseling Teacher” to enhance smaller models’ emotion support response abilities, significantly reducing the necessity of scaling up model size. To this end, we first introduce an iterative expansion framework, aiming to prompt the large teacher model to curate an expansive emotion support dialogue dataset. This curated dataset, termed ExTES, encompasses a broad spectrum of scenarios and is crafted with meticulous strategies …
Watme: Towards Lossless Watermarking Through Lexical Redundancy, Liang Chen, Yatao Bian, Yang Deng, Deng Cai, Shuaiyi Li, Peilin Zhao, Kam-Fai Wong
Watme: Towards Lossless Watermarking Through Lexical Redundancy, Liang Chen, Yatao Bian, Yang Deng, Deng Cai, Shuaiyi Li, Peilin Zhao, Kam-Fai Wong
Research Collection School Of Computing and Information Systems
Text watermarking has emerged as a pivotal technique for identifying machine-generated text. However, existing methods often rely on arbitrary vocabulary partitioning during decoding to embed watermarks, which compromises the availability of suitable tokens and significantly degrades the quality of responses. This study assesses the impact of watermarking on different capabilities of large language models (LLMs) from a cognitive science lens. Our finding highlights a significant disparity; knowledge recall and logical reasoning are more adversely affected than language generation. These results suggest a more profound effect of watermarking on LLMs than previously understood. To address these challenges, we introduce Watermarking with …
On The Multi-Turn Instruction Following For Conversational Web Agents, Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, Tat-Seng Chua
On The Multi-Turn Instruction Following For Conversational Web Agents, Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See-Kiong Ng, Tat-Seng Chua
Research Collection School Of Computing and Information Systems
Web agents powered by Large Language Models (LLMs) have demonstrated remarkable abilities in planning and executing multi-step interactions within complex web-based environments, fulfilling a wide range of web navigation tasks. Despite these advancements, the potential for LLM-powered agents to effectively engage with sequential user instructions in real-world scenarios has not been fully explored. In this work, we introduce a new task of Conversational Web Navigation, which necessitates sophisticated interactions that span multiple turns with both the users and the environment, supported by a specially developed dataset named Multi-Turn Mind2Web (MT-Mind2Web). To tackle the limited context length of LLMs and the …
Large Language Model Powered Agents For Information Retrieval, An Zhang, Yang Deng, Yankai Lin, Xu Chen, Ji-Rong Wen, Tat-Seng Chua
Large Language Model Powered Agents For Information Retrieval, An Zhang, Yang Deng, Yankai Lin, Xu Chen, Ji-Rong Wen, Tat-Seng Chua
Research Collection School Of Computing and Information Systems
The vital goal of information retrieval today extends beyond merely connecting users with relevant information they search for. It also aims to enrich the diversity, personalization, and interactivity of that connection, ensuring the information retrieval process is as seamless, beneficial, and supportive as possible in the global digital era. Current information retrieval systems often encounter challenges like a constrained understanding of queries, static and inflexible responses, limited personalization, and restricted interactivity. With the advent of large language models (LLMs), there's a transformative paradigm shift as we integrate LLM-powered agents into these systems. These agents bring forth crucial human capabilities like …
Let’S Think Outside The Box: Exploring Leap-Of-Thought In Large Language Models With Multimodal Humor Generation, Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, Pan Zhou
Let’S Think Outside The Box: Exploring Leap-Of-Thought In Large Language Models With Multimodal Humor Generation, Shanshan Zhong, Zhongzhan Huang, Shanghua Gao, Wushao Wen, Liang Lin, Marinka Zitnik, Pan Zhou
Research Collection School Of Computing and Information Systems
Chain-of-Thought (CoT) [2, 3] guides large language models (LLMs) to reason step-by-step, and can motivate their logical reasoning ability. While effective for logical tasks, CoT is not conducive to creative problem-solving which often requires out-of-box thoughts and is crucial for innovation advancements. In this paper, we explore the Leap-of-Thought (LoT) abilities within LLMs — a nonsequential, creative paradigm involving strong associations and knowledge leaps. To this end, we study LLMs on the popular Oogiri game which needs participants to have good creativity and strong associative thinking for responding unexpectedly and humorously to the given image, text, or both, and thus …
Enhancing Visual Grounding In Vision-Language Pre-Training With Position-Guided Text Prompts, Alex Jinpeng Wang, Pan Zhou, Mike Zheng Shou, Shuicheng Yan
Enhancing Visual Grounding In Vision-Language Pre-Training With Position-Guided Text Prompts, Alex Jinpeng Wang, Pan Zhou, Mike Zheng Shou, Shuicheng Yan
Research Collection School Of Computing and Information Systems
Vision-Language Pre-Training (VLP) has demonstrated remarkable potential in aligning image and text pairs, paving the way for a wide range of cross-modal learning tasks. Nevertheless, we have observed that VLP models often fall short in terms of visual grounding and localization capabilities, which are crucial for many downstream tasks, such as visual reasoning. In response, we introduce a novel Position-guided Text Prompt ( PTP ) paradigm to bolster the visual grounding abilities of cross-modal models trained with VLP. In the VLP phase, PTP divides an image into N x N blocks and employs a widely-used object detector to identify objects …
Large Language Models For Qualitative Research In Software Engineering: Exploring Opportunities And Challenges, Muneera Bano, Rashina Hoda, Didar Zowghi, Christoph Treude
Large Language Models For Qualitative Research In Software Engineering: Exploring Opportunities And Challenges, Muneera Bano, Rashina Hoda, Didar Zowghi, Christoph Treude
Research Collection School Of Computing and Information Systems
The recent surge in the integration of Large Language Models (LLMs) like ChatGPT into qualitative research in software engineering, much like in other professional domains, demands a closer inspection. This vision paper seeks to explore the opportunities of using LLMs in qualitative research to address many of its legacy challenges as well as potential new concerns and pitfalls arising from the use of LLMs. We share our vision for the evolving role of the qualitative researcher in the age of LLMs and contemplate how they may utilize LLMs at various stages of their research experience.
Compositional Policy Learning In Stochastic Control Systems With Formal Guarantees, Dorde Zikelic, Mathias Lechner, Abhinav Verma, Krishnendu Chatterjee, Thomas A. Henzinger
Compositional Policy Learning In Stochastic Control Systems With Formal Guarantees, Dorde Zikelic, Mathias Lechner, Abhinav Verma, Krishnendu Chatterjee, Thomas A. Henzinger
Research Collection School Of Computing and Information Systems
Reinforcement learning has shown promising results in learning neural network policies for complicated control tasks. However, the lack of formal guarantees about the behavior of such policies remains an impediment to their deployment. We propose a novel method for learning a composition of neural network policies in stochastic environments, along with a formal certificate which guarantees that a specification over the policy's behavior is satisfied with the desired probability. Unlike prior work on verifiable RL, our approach leverages the compositional nature of logical specifications provided in SPECTRL, to learn over graphs of probabilistic reach-avoid specifications. The formal guarantees are provided …
Unveiling Code Pre-Trained Models: Investigating Syntax And Semantics Capacities, Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhang Wang, Qiang Hu, Jie Zhang, Liu Yang
Unveiling Code Pre-Trained Models: Investigating Syntax And Semantics Capacities, Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhang Wang, Qiang Hu, Jie Zhang, Liu Yang
Research Collection School Of Computing and Information Systems
Code models have made significant advancements in code intelligence by encoding knowledge about programming languages. While previous studies have explored the capabilities of these models in learning code syntax, there has been limited investigation on their ability to understand code semantics. Additionally, existing analyses assume the number of edges between nodes at the abstract syntax tree (AST) is related to syntax distance, and also often require transforming the high-dimensional space of deep learning models to a low-dimensional one, which may introduce inaccuracies. To study how code models represent code syntax and semantics, we conduct a comprehensive analysis of 7 code …
Large Language Model Powered Agents In The Web, Yang Deng, An Zhang, Yankai Lin, Xu Chen, Ji-Rong Wen, Tat-Seng Chua
Large Language Model Powered Agents In The Web, Yang Deng, An Zhang, Yankai Lin, Xu Chen, Ji-Rong Wen, Tat-Seng Chua
Research Collection School Of Computing and Information Systems
Web applications serve as vital interfaces for users to access information, perform various tasks, and engage with content. Traditional web designs have predominantly focused on user interfaces and static experiences. With the advent of large language models (LLMs), there’s a paradigm shift as we integrate LLM-powered agents into these platforms. These agents bring forth crucial human capabilities like memory and planning to make them behave like humans in completing various tasks, effectively enhancing user engagement and offering tailored interactions in web applications. In this tutorial, we delve into the cutting-edge techniques of LLM-powered agents across various web applications, such as …
Plug-And-Play Policy Planner For Large Language Model Powered Dialogue Agents, Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, Tat-Seng Chua
Plug-And-Play Policy Planner For Large Language Model Powered Dialogue Agents, Yang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng, Tat-Seng Chua
Research Collection School Of Computing and Information Systems
Proactive dialogues serve as a practical yet challenging dialogue problem in the era of large language models (LLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs. Most existing studies enable the dialogue policy planning of LLMs using various prompting schemes or iteratively enhance this capability in handling the given case with verbal AI feedback. However, these approaches are either bounded by the policy planning capability of the frozen LLMs or hard to be transferred to new cases. In this work, we introduce a new dialogue policy planning paradigm to strategize LLMs for proactive dialogue …
Encoding Version History Context For Better Code Representation, Huy Nguyen, Christoph Treude, Patanamon Thongtanunam
Encoding Version History Context For Better Code Representation, Huy Nguyen, Christoph Treude, Patanamon Thongtanunam
Research Collection School Of Computing and Information Systems
With the exponential growth of AI tools that generate source code, understanding software has become crucial. When developers comprehend a program, they may refer to additional contexts to look for information, e.g. program documentation or historical code versions. Therefore, we argue that encoding this additional contextual information could also benefit code representation for deep learning. Recent papers incorporate contextual data (e.g. call hierarchy) into vector representation to address program comprehension problems. This motivates further studies to explore additional contexts, such as version history, to enhance models' understanding of programs. That is, insights from version history enable recognition of patterns in …
Large Language Model For Vulnerability Detection: Emerging Results And Future Directions, Xin Zhou, Ting Zhang, David Lo
Large Language Model For Vulnerability Detection: Emerging Results And Future Directions, Xin Zhou, Ting Zhang, David Lo
Research Collection School Of Computing and Information Systems
Previous learning-based vulnerability detection methods relied on either medium-sized pre-trained models or smaller neural networks from scratch. Recent advancements in Large Pre-Trained Language Models (LLMs) have showcased remarkable few-shot learning capabilities in various tasks. However, the effectiveness of LLMs in detecting software vulnerabilities is largely unexplored. This paper aims to bridge this gap by exploring how LLMs perform with various prompts, particularly focusing on two state-of-the-art LLMs: GPT-3.5 and GPT-4. Our experimental results showed that GPT-3.5 achieves competitive performance with the prior state-of-the-art vulnerability detection approach and GPT-4 consistently outperformed the state-of-the-art.
Code Search Is All You Need? Improving Code Suggestions With Code Search, Junkai Chen, Xing Hu, Zhenhao Li, Cuiyun Gao, Xin Xia, David Lo
Code Search Is All You Need? Improving Code Suggestions With Code Search, Junkai Chen, Xing Hu, Zhenhao Li, Cuiyun Gao, Xin Xia, David Lo
Research Collection School Of Computing and Information Systems
Modern integrated development environments (IDEs) provide various automated code suggestion techniques (e.g., code completion and code generation) to help developers improve their efficiency. Such techniques may retrieve similar code snippets from the code base or leverage deep learning models to provide code suggestions. However, how to effectively enhance the code suggestions using code retrieval has not been systematically investigated. In this paper, we study and explore a retrieval-augmented framework for code suggestions. Specifically, our framework leverages different retrieval approaches and search strategies to search similar code snippets. Then the retrieved code is used to further enhance the performance of language …
Greening Large Language Models Of Code, Jieke Shi, Zhou Yang, Hong Jin Kang, Bowen Xu, Junda He, David Lo
Greening Large Language Models Of Code, Jieke Shi, Zhou Yang, Hong Jin Kang, Bowen Xu, Junda He, David Lo
Research Collection School Of Computing and Information Systems
Large language models of code have shown remarkable effectiveness across various software engineering tasks. Despite the availability of many cloud services built upon these powerful models, there remain several scenarios where developers cannot take full advantage of them, stemming from factors such as restricted or unreliable internet access, institutional privacy policies that prohibit external transmission of code to third-party vendors, and more. Therefore, developing a compact, efficient, and yet energy-saving model for deployment on developers' devices becomes essential.To this aim, we propose Avatar, a novel approach that crafts a deployable model from a large language model of code by optimizing …
Unveiling Memorization In Code Models, Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, David Lo
Unveiling Memorization In Code Models, Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsun Kim, Donggyun Han, David Lo
Research Collection School Of Computing and Information Systems
The availability of large-scale datasets, advanced architectures, and powerful computational resources have led to effective code models that automate diverse software engineering activities. The datasets usually consist of billions of lines of code from both open-source and private repositories. A code model memorizes and produces source code verbatim, which potentially contains vulnerabilities, sensitive information, or code with strict licenses, leading to potential security and privacy issues.This paper investigates an important problem: to what extent do code models memorize their training data? We conduct an empirical study to explore memorization in large pre-trained code models. Our study highlights that simply extracting …
Streamlining Java Programming: Uncovering Well-Formed Idioms With Idiomine, Yanming Yang, Xing Hu, Xin Xia, David Lo, Xiaohu Yang
Streamlining Java Programming: Uncovering Well-Formed Idioms With Idiomine, Yanming Yang, Xing Hu, Xin Xia, David Lo, Xiaohu Yang
Research Collection School Of Computing and Information Systems
Code idioms are commonly used patterns, techniques, or practices that aid in solving particular problems or specific tasks across multiple software projects. They can improve code quality, performance, and maintainability, and also promote program standardization and reuse across projects. However, identifying code idioms is significantly challenging, as existing studies have still suffered from three main limitations. First, it is difficult to recognize idioms that span non-contiguous code lines. Second, identifying idioms with intricate data flow and code structures can be challenging. Moreover, they only extract dataset-specific idioms, so common idioms or well-established code/design patterns that are rarely found in datasets …
Fixing Your Own Smells: Adding A Mistake-Based Familiarization Step When Teaching Code Refactoring, Ivan Wei Han Tan, Christopher M. Poskitt
Fixing Your Own Smells: Adding A Mistake-Based Familiarization Step When Teaching Code Refactoring, Ivan Wei Han Tan, Christopher M. Poskitt
Research Collection School Of Computing and Information Systems
Programming problems can be solved in a multitude of functionally correct ways, but the quality of these solutions (e.g. readability, maintainability) can vary immensely. When code quality is poor, symptoms emerge in the form of 'code smells', which are specific negative characteristics (e.g. duplicate code) that can be resolved by applying refactoring patterns. Many undergraduate computing curricula train students on this software engineering practice, often doing so via exercises on unfamiliar instructor-provided code. Our observation, however, is that this makes it harder for novices to internalise refactoring as part of their own development practices. In this paper, we propose a …