Open Access. Powered by Scholars. Published by Universities.®

Digital Commons Network™

Open Access. Powered by Scholars. Published by Universities.®

Computer Sciences

Institution
Keyword
Publication Year
Publication
Publication Type
File Type

Articles 1231 - 1260 of 4474

Full-Text Articles in Entire DC Network

Large Language Models As Source Planner For Personalized Knowledge-Grounded Dialogues, Hongru Wang, Minda Hu, Yang Deng, Rui Wang, Fei Mi, Weichao Wang, Yasheng Wang, Wai-Chung Kwan, Irwin King, Kam-Fai Wong Dec 2023

Large Language Models As Source Planner For Personalized Knowledge-Grounded Dialogues, Hongru Wang, Minda Hu, Yang Deng, Rui Wang, Fei Mi, Weichao Wang, Yasheng Wang, Wai-Chung Kwan, Irwin King, Kam-Fai Wong

Research Collection School Of Computing and Information Systems

Open-domain dialogue system usually requires different sources of knowledge to generate more informative and evidential responses. However, existing knowledge-grounded dialogue systems either focus on a single knowledge source or overlook the dependency between multiple sources of knowledge, which may result in generating inconsistent or even paradoxical responses. To incorporate multiple knowledge sources and dependencies between them, we propose SAFARI, a novel framework that leverages the exceptional capabilities of large language models (LLMs) in planning, understanding, and incorporating under both supervised and unsupervised settings. Specifically, SAFARI decouples the knowledge grounding into multiple sources and response generation, which allows easy extension to …

Go to article

Llm-Adapters: An Adapter Family For Parameter-Efficient Fine-Tuning Of Large Language Models, Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Ka-Wei Lee Dec 2023

Llm-Adapters: An Adapter Family For Parameter-Efficient Fine-Tuning Of Large Language Models, Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, Roy Ka-Wei Lee

Research Collection School Of Computing and Information Systems

The success of large language models (LLMs), like GPT-4 and ChatGPT, has led to the development of numerous cost-effective and accessible alternatives that are created by finetuning open-access LLMs with task-specific data (e.g., ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly one of the most attractive topics, as it only requires fine-tuning a few external parameters instead of the entire LLMs while achieving comparable or even better performance. To enable further research on PEFT methods of LLMs, this paper presents LLMAdapters, an easy-to-use framework that integrates various adapters into LLMs and …

Go to article

Rome: Evaluating Pre-Trained Vision-Language Models On Reasoning Beyond Visual Common Sense, Kankan Zhou, Eason Lai, Au Wei Bin Yeong, Kyriakos Mouratidis, Jing Jiang Dec 2023

Rome: Evaluating Pre-Trained Vision-Language Models On Reasoning Beyond Visual Common Sense, Kankan Zhou, Eason Lai, Au Wei Bin Yeong, Kyriakos Mouratidis, Jing Jiang

Research Collection School Of Computing and Information Systems

Humans possess a strong capability for reasoning beyond common sense. For example, given an unconventional image of a goldfish laying on the table next to an empty fishbowl, a human would effortlessly determine that the fish is not inside the fishbowl. The case, however, may be different for a vision-language model, whose reasoning could gravitate towards the common scenario that the fish is inside the bowl, despite the visual input. In this paper, we introduce a novel probing dataset named ROME (reasoning beyond commonsense knowledge) to evaluate whether the state-of-the-art pre-trained vision-language models have the reasoning capability to correctly interpret …

Go to article

Robust Prompt Optimization For Large Language Models Against Distribution Shifts, Moxin Li, Wenjie Wang, Fuli Feng, Yixin Cao, Jizhi Zhang, Tat-Seng Chua Dec 2023

Robust Prompt Optimization For Large Language Models Against Distribution Shifts, Moxin Li, Wenjie Wang, Fuli Feng, Yixin Cao, Jizhi Zhang, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Large Language Model (LLM) has demonstrated significant ability in various Natural Language Processing tasks. However, their effectiveness is highly dependent on the phrasing of the task prompt, leading to research on automatic prompt optimization using labeled task data. We reveal that these prompt optimization techniques are vulnerable to distribution shifts such as subpopulation shifts, which are common for LLMs in real-world scenarios such as customer reviews analysis. In this light, we propose a new problem of robust prompt optimization for LLMs against distribution shifts, which requires the prompt optimized over the labeled source group can simultaneously generalize to an unlabeled …

Go to article

Molca: Molecular Graph-Language Modeling With Cross-Modal Projector And Uni-Modal Adapter, Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, Tat-Seng Chua Dec 2023

Molca: Molecular Graph-Language Modeling With Cross-Modal Projector And Uni-Modal Adapter, Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Language Models (LMs) have demonstrated impressive molecule understanding ability on various 1D text-related tasks. However, they inherently lack 2D graph perception — a critical ability of human professionals in comprehending molecules’ topological structures. To bridge this gap, we propose MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter. MolCA enables an LM (i.e., Galactica) to understand both text- and graph-based molecular contents via the cross-modal projector. Specifically, the cross-modal projector is implemented as a QFormer to connect a graph encoder’s representation space and an LM’s text space. Further, MolCA employs a uni-modal adapter (i.e., LoRA) for the LM’s efficient …

Go to article

Wsdms: Debunk Fake News Via Weakly Supervised Detection Of Misinforming Sentences With Contextualized Social Wisdom, Ruichao Yang, Wei Gao, Jing Ma, Hongzhan Lin, Zhiwei Yang Dec 2023

Wsdms: Debunk Fake News Via Weakly Supervised Detection Of Misinforming Sentences With Contextualized Social Wisdom, Ruichao Yang, Wei Gao, Jing Ma, Hongzhan Lin, Zhiwei Yang

Research Collection School Of Computing and Information Systems

In recent years, we witness the explosion of false and unconfirmed information (i.e., rumors) that went viral on social media and shocked the public. Rumors can trigger versatile, mostly controversial stance expressions among social media users. Rumor verification and stance detection are different yet relevant tasks. Fake news debunking primarily focuses on determining the truthfulness of news articles, which oversimplifies the issue as fake news often combines elements of both truth and falsehood. Thus, it becomes crucial to identify specific instances of misinformation within the articles. In this research, we investigate a novel task in the field of fake news …

Go to article

Disentangling Transformer Language Models As Superposed Topic Models, Jia Peng Lim, Hady Wirawan Lauw Dec 2023

Disentangling Transformer Language Models As Superposed Topic Models, Jia Peng Lim, Hady Wirawan Lauw

Research Collection School Of Computing and Information Systems

Topic Modelling is an established research area where the quality of a given topic is measured using coherence metrics. Often, we infer topics from Neural Topic Models (NTM) by interpreting their decoder weights, consisting of top-activated words projected from individual neurons. Transformer-based Language Models (TLM) similarly consist of decoder weights. However, due to its hypothesised superposition properties, the final logits originating from the residual path are considered uninterpretable. Therefore, we posit that we can interpret TLM as superposed NTM by proposing a novel weight-based, model-agnostic and corpus-agnostic approach to search and disentangle decoder-only TLM, potentially mapping individual neurons to multiple …

Go to article

Transformer-Based Multi-Task Learning For Crisis Actionability Extraction, Yuhao Zhang, Siaw Ling Lo, Phyo Yi Win Myint Dec 2023

Transformer-Based Multi-Task Learning For Crisis Actionability Extraction, Yuhao Zhang, Siaw Ling Lo, Phyo Yi Win Myint

Research Collection School Of Computing and Information Systems

Social media has become a valuable information source for crisis informatics. While various methods were proposed to extract relevant information during a crisis, their adoption by field practitioners remains low. In recent fieldwork, actionable information was identified as the primary information need for crisis responders and a key component in bridging the significant gap in existing crisis management tools. In this paper, we proposed a Crisis Actionability Extraction System for filtering, classification, phrase extraction, severity estimation, localization, and aggregation of actionable information altogether. We examined the effectiveness of transformer-based LSTM-CRF architecture in Twitter-related sequence tagging tasks and simultaneously extracted actionable …

Go to article

Prompting And Evaluating Large Language Models For Proactive Dialogues: Clarification, Target-Guided, And Non-Collaboration, Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, Tat-Seng Chua Dec 2023

Prompting And Evaluating Large Language Models For Proactive Dialogues: Clarification, Target-Guided, And Non-Collaboration, Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Conversational systems based on Large Language Models (LLMs), such as ChatGPT, show exceptional proficiency in context understanding and response generation. However, they still possess limitations, such as failing to ask clarifying questions to ambiguous queries or refuse users' unreasonable requests, both of which are considered as key aspects of a conversational agent's proactivity. This raises the question of whether LLM-based conversational systems are equipped to handle proactive dialogue problems. In this work, we conduct a comprehensive analysis of LLM-based conversational systems, specifically focusing on three key aspects of proactive dialogues: clarification, target-guided, and non-collaborative dialogues. To trigger the proactivity of …

Go to article

Attack Prompt Generation For Red Teaming And Defending Large Language Models, Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, Xiangnan He Dec 2023

Attack Prompt Generation For Red Teaming And Defending Large Language Models, Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, Xiangnan He

Research Collection School Of Computing and Information Systems

Large language models (LLMs) are susceptible to red teaming attacks, which can induce LLMs to generate harmful content. Previous research constructs attack prompts via manual or automatic methods, which have their own limitations on construction cost and quality. To address these issues, we propose an integrated approach that combines manual and automatic methods to economically generate high-quality attack prompts. Specifically, considering the impressive capabilities of newly emerged LLMs, we propose an attack framework to instruct LLMs to mimic human-generated prompts through in-context learning. Furthermore, we propose a defense framework that fine-tunes victim LLMs through iterative interactions with the attack framework …

Go to article

Depwignn: A Depth-Wise Graph Neural Network For Multi-Hop Spatial Reasoning In Text, Shuaiyi Li, Yang Deng, Wai Lam Dec 2023

Depwignn: A Depth-Wise Graph Neural Network For Multi-Hop Spatial Reasoning In Text, Shuaiyi Li, Yang Deng, Wai Lam

Research Collection School Of Computing and Information Systems

Spatial reasoning in text plays a crucial role in various real-world applications. Existing approaches for spatial reasoning typically infer spatial relations from pure text, which overlook the gap between natural language and symbolic structures. Graph neural networks (GNNs) have showcased exceptional proficiency in inducing and aggregating symbolic structures. However, classical GNNs face challenges in handling multi-hop spatial reasoning due to the over-smoothing issue, i.e., the performance decreases substantially as the number of graph layers increases. To cope with these challenges, we propose a novel Depth-Wise Graph Neural Network (DepWiGNN). Specifically, we design a novel node memory scheme and aggregate the …

Go to article

Unifying Text, Tables, And Images For Multimodal Question Answering, Haohao Luo, Ying Shen, Yang Deng Dec 2023

Unifying Text, Tables, And Images For Multimodal Question Answering, Haohao Luo, Ying Shen, Yang Deng

Research Collection School Of Computing and Information Systems

Multimodal question answering (MMQA), which aims to derive the answer from multiple knowledge modalities (e.g., text, tables, and images), has received increasing attention due to its board applications. Current approaches to MMQA often rely on single-modal or bi-modal QA models, which limits their ability to effectively integrate information across all modalities and leverage the power of pre-trained language models. To address these limitations, we propose a novel framework called UniMMQA, which unifies three different input modalities into a text-to-text format by employing position-enhanced table linearization and diversified image captioning techniques. Additionally, we enhance cross-modal reasoning by incorporating a multimodal rationale …

Go to article

Supporting Software Engineers With Large Language Model-Based Automation, Ting Zhang Dec 2023

Supporting Software Engineers With Large Language Model-Based Automation, Ting Zhang

Dissertations and Theses Collection (Open Access)

In recent years, software engineering (SE) has witnessed significant growth, leading to the creation and sharing of an abundance of software artifacts such as source code, bug reports, and pull requests. Analyzing these artifacts is crucial for comprehending the sentiments of software developers and automating various SE tasks, ultimately leading to more human-centered automated SE and enhancing software development efficiency. However, the diverse and unstructured nature of software text poses a significant challenge to this analysis. In response, researchers have investigated a variety of approaches, including the utilization of natural language processing techniques. The advent of large language models (LLMs), …

Go to article

Enhanced Content-Based Fake News Detection Methods With Context-Labeled News Sources, Duncan Arnfield Dec 2023

Enhanced Content-Based Fake News Detection Methods With Context-Labeled News Sources, Duncan Arnfield

Electronic Theses and Dissertations

This work examined the relative effectiveness of multilayer perceptron, random forest, and multinomial naïve Bayes classifiers, trained using bag of words and term frequency-inverse dense frequency transformations of documents in the Fake News Corpus and Fake and Real News Dataset. The goal of this work was to help meet the formidable challenges posed by proliferation of fake news to society, including the erosion of public trust, disruption of social harmony, and endangerment of lives. This training included the use of context-categorized fake news in an effort to enhance the tools’ effectiveness. It was found that term frequency-inverse dense frequency provided …

Go to article

Review Classification Using Natural Language Processing And Deep Learning, Brian Nazareth Dec 2023

Review Classification Using Natural Language Processing And Deep Learning, Brian Nazareth

Electronic Theses, Projects, and Dissertations

Sentiment Analysis is an ongoing research in the field of Natural Language Processing (NLP). In this project, I will evaluate my testing against an Amazon Reviews Dataset, which contains more than 100 thousand reviews from customers. This project classifies the reviews using three methods – using a sentiment score by comparing the words of the reviews based on every positive and negative word that appears in the text with the Opinion Lexicon dataset, by considering the text’s variating sentiment polarity scores with a Python library called TextBlob, and with the help of neural network training. I have created a neural …

Go to article

Towards A Model Of The Mapping Between English And Spanish Prosody, Jonathan Avila Dec 2023

Towards A Model Of The Mapping Between English And Spanish Prosody, Jonathan Avila

Open Access Theses & Dissertations

Current speech-to-speech translation systems face challenges in effectively translating the nuances of prosody, which plays a pivotal role in conveying speaker intent and stance in dialog. This limitation restricts cross-lingual communication, especially in situations demanding deeper interpersonal understanding. To address this, this research delves into the relationships between prosody and its pragmatic functions, in English and Spanish. First, I discuss a data collection protocol in which bilingual speakers re-enact utterances from an earlier conversation in their other language, then describe an English-Spanish corpus, consisting of 3816 matched utterance pairs. Second, I describe a prosodic dissimilarity metric based on Euclidean distance …

Go to article

The Persuasive Effect Of Ai-Synthesized Voices, Hannah H. Chang, Anirban Mukherjee Dec 2023

The Persuasive Effect Of Ai-Synthesized Voices, Hannah H. Chang, Anirban Mukherjee

Research Collection Lee Kong Chian School Of Business

Artificial intelligence (AI) technology seeks to emulate humans. One aspect is AI-synthesized voices, used in voice assistants (such as Amazon Alexa, Apple Siri, and Google Assistant) to assistive technologies (such as voiceover narration in product videos). For example, there are currently more than 3.25 billion voice assistants; a number that is expected to touch about 8 billion by next year (i.e., 2023) (Statista 2022). With the extensive availability and enhanced accuracy of AI-synthesized voices, consumer research is starting to examine the impact of AI-synthesized voices on consumer information processing and decision making. The extant literature, however, is relatively limited because …

Go to article

Beyond Factuality: A Comprehensive Evaluation Of Large Language Models As Knowledge Generators, Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, Kam-Fai Wong Dec 2023

Beyond Factuality: A Comprehensive Evaluation Of Large Language Models As Knowledge Generators, Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, Kam-Fai Wong

Research Collection School Of Computing and Information Systems

Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. Yet, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. In light of this, we introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to systematically and automatically evaluate generated knowledge from six important perspectives - Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely-studied knowledge-intensive tasks, i.e., open-domain question answering and knowledge-grounded dialogue. Surprisingly, our study reveals that the …

Go to article

Customer Cybersecurity And Supplier Cost Management Strategy, Xu Yang, Peng Liang, Nan Hu, Fujing Xue Dec 2023

Customer Cybersecurity And Supplier Cost Management Strategy, Xu Yang, Peng Liang, Nan Hu, Fujing Xue

Research Collection School Of Computing and Information Systems

In this paper, we explore the spillover effect of customer firms’ data breaches on their upstream supplier firms’ cost management strategies, proxied by cost stickiness. Our primary analyses suggest that data breaches suffered by customer firms are associated with a decrease in cost stickiness among supplier firms. Furthermore, the reductions in supplier cost stickiness are stronger if suppliers are managed by CEOs from national cultural groups with high uncertainty avoidance, low long-term orientations, and/or low individualism. In sum, the findings contribute to both Information Systems (IS) and Operations Management (OM) disciplines in terms of data breach, cost management strategy, and …

Go to article

Covariance-Based Causal Debiasing For Entity And Relation Extraction, Lin Ren, Yongbin Liu, Yixin Cao, Chunping Ouyang Dec 2023

Covariance-Based Causal Debiasing For Entity And Relation Extraction, Lin Ren, Yongbin Liu, Yixin Cao, Chunping Ouyang

Research Collection School Of Computing and Information Systems

Joint entity and relation extraction tasks aim to recognize named entities and extract relations simultaneously. Suffering from a variety of data biases, such as data selection bias, and distribution bias (out of distribution, long-tail distribution), serious concerns can be witnessed to threaten the model’s transferability, robustness, and generalization. In this work, we address the above problems from a causality perspective. We propose a novel causal framework called covariance and variance optimization framework (OVO) to optimize feature representations and conduct general debiasing. In particular, the proposed covariance optimizing (COP) minimizes characterizing features’ covariance for alleviating the selection and distribution bias and …

Go to article

A Comprehensive Evaluation Of Large Language Models On Legal Judgment Prediction, Ruihao Shui, Yixin Cao, Xiang Wang, Tat-Seng Chua Dec 2023

A Comprehensive Evaluation Of Large Language Models On Legal Judgment Prediction, Ruihao Shui, Yixin Cao, Xiang Wang, Tat-Seng Chua

Research Collection School Of Computing and Information Systems

Large language models (LLMs) have demonstrated great potential for domain-specific applications, such as the law domain. However, recent disputes over GPT-4’s law evaluation raise questions concerning their performance in real-world legal tasks. To systematically investigate their competency in the law, we design practical baseline solutions based on LLMs and test on the task of legal judgment prediction. In our solutions, LLMs can work alone to answer open questions or coordinate with an information retrieval (IR) system to learn from similar cases or solve simplified multi-choice questions. We show that similar cases and multi-choice options, namely label candidates, included in prompts …

Go to article

A Black-Box Attack On Code Models Via Representation Nearest Neighbor Search, Jie Zhang, Wei Ma, Qiang Hu, Shangqing Liu, Xiaofei Xie, Yves Le Traon, Yang Liu Dec 2023

A Black-Box Attack On Code Models Via Representation Nearest Neighbor Search, Jie Zhang, Wei Ma, Qiang Hu, Shangqing Liu, Xiaofei Xie, Yves Le Traon, Yang Liu

Research Collection School Of Computing and Information Systems

Existing methods for generating adversarial code examples face several challenges: limted availability of substitute variables, high verification costs for these substitutes, and the creation of adversarial samples with noticeable perturbations. To address these concerns, our proposed approach, RNNS, uses a search seed based on historical attacks to find potential adversarial substitutes. Rather than directly using the discrete substitutes, they are mapped to a continuous vector space using a pre-trained variable name encoder. Based on the vector representation, RNNS predicts and selects better substitutes for attacks. We evaluated the performance of RNNS across six coding tasks encompassing three programming languages: Java, …

Go to article

An Exhaustive Review Of Neutrosophic Logic In Addressing Image Processing Issues, Samia Mandour Dec 2023

An Exhaustive Review Of Neutrosophic Logic In Addressing Image Processing Issues, Samia Mandour

Neutrosophic Systems with Applications

Since the importance of images in our lives and the advancements in computer data gathering methods, anyone can collect a large number of images, but most of them cannot be processed manually. Image processing therefore becomes appealing since various types of data may be represented and processed digitally. Image processing has become the most popular processing method, employed in security camera films, healthcare images, images from remote sensors, and naturalistic image/videos because of fast computers and processors. In order to raise cognitive function and speed up decision-making, image processing is crucial to many information access systems. Since ambiguity now permeates …

Go to article

Emotional Health And Climate-Change-Related Stressor Extraction From Social Media: A Case Study Using Hurricane Harvey, Thanh Bui, Andrea Hannah, Sanjay Madria, Rosemary Nabaweesi, Eugene Levin, Michael Wilson, Long Nguyen Dec 2023

Emotional Health And Climate-Change-Related Stressor Extraction From Social Media: A Case Study Using Hurricane Harvey, Thanh Bui, Andrea Hannah, Sanjay Madria, Rosemary Nabaweesi, Eugene Levin, Michael Wilson, Long Nguyen

Computer Science Faculty Research & Creative Works

Climate change has led to a variety of disasters that have caused damage to infrastructure and the economy with societal impacts to human living. Understanding people's emotions and stressors during disaster times will enable preparation strategies for mitigating further consequences. in this paper, we mine emotions and stressors encountered by people and shared on Twitter during Hurricane Harvey in 2017 as a showcase. in this work, we acquired a dataset of tweets from Twitter on Hurricane Harvey from 20 August 2017 to 30 August 2017. the dataset consists of around 400,000 tweets and is available on Kaggle. Next, a BERT-Based …

Go to article

Llm4vis: Explainable Visualization Recommendation Using Chatgpt, Lei Wang, Songheng Zhang, Yun Wang, Ee-Peng Lim, Yong Wang Dec 2023

Llm4vis: Explainable Visualization Recommendation Using Chatgpt, Lei Wang, Songheng Zhang, Yun Wang, Ee-Peng Lim, Yong Wang

Research Collection School Of Computing and Information Systems

Data visualization is a powerful tool for exploring and communicating insights in various domains. To automate visualization choice for datasets, a task known as visualization recommendation has been proposed. Various machine-learning-based approaches have been developed for this purpose, but they often require a large corpus of dataset-visualization pairs for training and lack natural explanations for their results. To address this research gap, we propose LLM4Vis, a novel ChatGPT-based prompting approach to perform visualization recommendation and return human-like explanations using very few demonstration examples. Our approach involves feature description, demonstration example selection, explanation generation, demonstration example construction, and inference steps. To …

Go to article

Large Language Model Is Not A Good Few-Shot Information Extractor, But A Good Reranker For Hard Samples!, Yubo Ma, Yixin Cao, Yongchin Hong, Aixin Sun Dec 2023

Large Language Model Is Not A Good Few-Shot Information Extractor, But A Good Reranker For Hard Samples!, Yubo Ma, Yixin Cao, Yongchin Hong, Aixin Sun

Research Collection School Of Computing and Information Systems

Large Language Models (LLMs) have made remarkable strides in various tasks. However, whether they are competitive few-shot solvers for information extraction (IE) tasks and surpass fine-tuned small Pre-trained Language Models (SLMs) remains an open problem. This paper aims to provide a thorough answer to this problem, and moreover, to explore an approach towards effective and economical IE systems that combine the strengths of LLMs and SLMs. Through extensive experiments on nine datasets across four IE tasks, we show that LLMs are not effective few-shot information extractors in general, given their unsatisfactory performance in most settings and the high latency and …

Go to article

Benchmarking Foundation Models With Language-Model-As-An-Examiner, Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, Lei Hou Dec 2023

Benchmarking Foundation Models With Language-Model-As-An-Examiner, Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, Lei Hou

Research Collection School Of Computing and Information Systems

Numerous benchmarks have been established to assess the performance of foundation models on open-ended question answering, which serves as a comprehensive test of a model’s ability to understand and generate language in a manner similar to humans. Most of these works focus on proposing new datasets, however, we see two main issues within previous benchmarking pipelines, namely testing leakage and evaluation automation. In this paper, we propose a novel benchmarking framework, Language-Model-as-an-Examiner, where the LM serves as a knowledgeable examiner that formulates questions based on its knowledge and evaluates responses in a reference-free manner. Our framework allows for effortless extensibility …

Go to article

Knowledge Graph Enhanced Aspect-Based Sentiment Analysis Incorporating External Knowledge, Autumn Teo, Zhaoxia Wang, Haibo Pen, Budhitama Subagdja, Seng-Beng Ho, Boon Kiat Quek Dec 2023

Knowledge Graph Enhanced Aspect-Based Sentiment Analysis Incorporating External Knowledge, Autumn Teo, Zhaoxia Wang, Haibo Pen, Budhitama Subagdja, Seng-Beng Ho, Boon Kiat Quek

Research Collection School Of Computing and Information Systems

Aspect-based sentiment analysis (ABSA) is a fine-grained task of sentiment analysis. To better comprehend long complicated sentences and obtain accurate aspect-specific information, linguistic and commonsense knowledge are generally required in this task. However, most current methods employ complicated and inefficient approaches to incorporate external knowledge, e.g., directly searching the graph nodes. Additionally, the complementarity between external knowledge and linguistic information has not been thoroughly studied. To this end, we propose a knowledge graph augmented network (KGAN), which aims to effectively incorporate external knowledge with explicitly syntactic and contextual information. In particular, KGAN captures the sentiment feature representations from multiple different …

Go to article

Just Adjust One Prompt: Enhancing In-Context Dialogue Scoring Via Constructing The Optimal Subgraph Of Demonstrations And Prompts, Jiashu Pu, Ling Cheng, Lu Fan, Tangjie Lv, Rongsheng Zhang Dec 2023

Just Adjust One Prompt: Enhancing In-Context Dialogue Scoring Via Constructing The Optimal Subgraph Of Demonstrations And Prompts, Jiashu Pu, Ling Cheng, Lu Fan, Tangjie Lv, Rongsheng Zhang

Research Collection School Of Computing and Information Systems

The use of modern Large Language Models (LLMs) as chatbots still has some problems such as hallucinations and lack of empathy. Identifying these issues can help improve chatbot performance. The community has been continually iterating on reference-free dialogue evaluation methods based on large language models (LLMs) that can be readily applied. However, many of these LLM-based metrics require selecting specific datasets and developing specialized training tasks for different evaluation dimensions (e.g., coherence, informative). The developing step can be time-consuming and may need to be repeated for new evaluation dimensions. To enable efficient and flexible adaptation to diverse needs of dialogue …

Go to article

Do Contributing Files Provide Information About Oss Newcomers' Onboarding Barriers?, Felipe Fronchetti, David Shepherd, Igor Wiese, Christoph Treude, Marco Gerosa, Igor Steinmacher Dec 2023

Do Contributing Files Provide Information About Oss Newcomers' Onboarding Barriers?, Felipe Fronchetti, David Shepherd, Igor Wiese, Christoph Treude, Marco Gerosa, Igor Steinmacher

Research Collection School Of Computing and Information Systems

Effectively onboarding newcomers is essential for the success of open source projects. These projects often provide onboarding guidelines in their ‘CONTRIBUTING’ files (e.g., CONTRIBUTING.md on GitHub). These files explain, for example, how to find open tasks, implement solutions, and submit code for review. However, these files often do not follow a standard structure, can be too large, and miss barriers commonly found by newcomers. In this paper, we propose an automated approach to parse these CONTRIBUTING files and assess how they address onboarding barriers. We manually classified a sample of files according to a model of onboarding barriers from the …

Go to article

First Prev Next Last