Artificial Intelligence and Robotics Commons^™

Quantifying The Role Of Active Listening And Reassurance In Virtual Health Coach Interactions, Ghulam Hussain, Brian Keegan, Robert Ross

Conference papers

Conversational Agents have the potential to support healthcare through coaching exercise routines, but are still lacking in demonstrating authentic social behaviours to support engagement. To this end, we present a series of experiments that we conducted in order to investigate how automated health care coaches can be more effective when their interaction style is tailored to demonstrate qualities associated with a good bedside manner, namely active listening and reassurance. To test this, we first developed a dataset of 135 dialogue excerpts from three distinct sources, i.e., original, handcrafted and LLMs, the latter two of which were tuned to demonstrate specific …

On The Benefits Of Directness In Virtual Characters For Motivational Interviews, Michael O'Mahony, Cathy Ennis, Robert Ross

Conference papers

Understanding the factors influencing successful engagement with Embodied Conversational Agents (ECAs) remains a significant challenge. This understanding could be used to personalise agents to users to improve interactions. Some studies have shown that simulating personalities in healthcare agents can improve effectiveness and engagement. However, it is not yet well understood how variations of agent personality can be leveraged to improve user engagement with Motivational Interviewing (MI) ECAs. Specifically how the balance between agent warmth and directness can be controlled in an MI agent to improve likeability and engagement. We conducted an online Wizard-of-Oz (WoZ) mediated study of two variants of …

Habit Coach: Customising Rag-Based Chatbots To Support Behavior Change, Arian Fooroogh Mand Arabi, Cansu Koyuturk, Michael O'Mahony, Raffaella Calati, Dimitri Ognibene

Conference papers

This paper presents the iterative development of Habit Coach, a GPT-based chatbot designed to support users in habit change through personalized interaction. Employing a user-centered design approach, we developed the chatbot using a Retrieval-Augmented Generation (RAG) system, which enables behavior personalization without retraining the underlying language model (GPT-4). The system leverages document retrieval and specialized prompts to tailor interactions, drawing from Cognitive Behavioral Therapy (CBT) and narrative therapy techniques. A key challenge in the development process was the difficulty of translating declarative knowledge into effective interaction behaviors. In the initial phase, the chatbot was provided with declarative knowledge about CBT …

Getting To The Point: Contrasting Directness And Warmth In Motivational Embodied Conversational Agents, Michael O'Mahony, Cathy Ennis, Robert Ross

Conference papers

Enhancing long-term engagement with conversational agents remains a significant challenge. Controlling the perceived warmth or directness of an agent’s personality through the style of its generated text could be used to increase user likeability. This paper reports an investigation of a Wizard-of-Oz (WoZ) mediated study of two variants of a motivational embodied conversational agent to measure user perception of and attitudes towards warmth in interaction style. Results show a significant effect of users preferring an agent with a "more direct" personality for this scenario, though this effect is in many ways nuanced.

Human Centered Approaches And Taxonomies For Explainable Artificial Intelligence, Helen Sheridan, Emma Murphy, Dympna O'Sullivan

Conference papers

Recent interest within the research community related to explainable artificial intelligence (XAI) has led to a profuse amount of literature on the subject. Those who wish to tackle the domain from an HCI focus may be presented with overwhelming material, most of which does not pertain to human aspects of XAI. Taxonomies can serve to categorize a subject into topic areas and distill content into an overview of the field. This late breaking work intends to help those within the HCI community with a focus on XAI to understand relevant aspects of human centered XAI. We also present a taxonomy …

Using Chatgpt To Generate Gendered Language, Shweta Soundararajan, Manuela Nayantara Jeyaraj, Sarah Jane Delany

Conference papers

Gendered language is the use of words that denote an individual's gender. This can be explicit where the gender is evident in the actual word used, e.g. mother, she, man, but it can also be implicit where social roles or behaviours can signal an individual's gender - for example, expectations that women display communal traits (e.g., affectionate, caring, gentle) and men display agentic traits (e.g., assertive, competitive, decisive). The use of gendered language in NLP systems can perpetuate gender stereotypes and bias. This paper proposes an approach to generating gendered language datasets using ChatGPT which will provide data for data-driven …

A Deep Learning-Based Object Detection Framework For Automatic Asphalt Pavement Patch Detection Using Laser Profiling Images, Ibrahim Hassan Syed, Susan Mckeever Dr., Kieran Feighan, David Power, Dympna O'Sullivan

Conference papers

Road maintenance and the early detection of road defects rely on routine pavement inspections. While advanced 3D laser profiling systems have the capability to automatically identify certain types of distress such as cracks and ruts, more complex pavement damage, including patches, often require manual identification. To address this limitation, this study proposes an automated patch detection system that employs object detection techniques. The results demonstrate the ability of object detection models to accurately identify patches in laser profiling images, indicating that the proposed approach has the capability to significantly enhance automation in visual inspection processes. This has the potential for …

Wifi-Based Human Activity Recognition Using Attention-Based Bilstm, Amany Elkelany, Robert J. Ross, Susan Mckeever

Conference papers

Recently, significant efforts have been made to explore human activity recognition (HAR) techniques that use information gathered by existing indoor wireless infrastructures through WiFi signals without demanding the monitored subject to carry a dedicated device. The key intuition is that different activities introduce different multi-paths in WiFi signals and generate different patterns in the time series of channel state information (CSI). In this paper, we propose and evaluate a full pipeline for a CSI-based human activity recognition framework for 12 activities in three different spatial environments using two deep learning models: ABiLSTM and CNN-ABiLSTM. Evaluation experiments have demonstrated that the …

Determining Child Sexual Abuse Posts Based On Artificial Intelligence, Susan Mckeever, Christina Thorpe, Vuong Ngo

Conference papers

The volume of child sexual abuse materials (CSAM) created and shared daily both surface web platforms such as Twitter and dark web forums is very high. Based on volume, it is not viable for human experts to intercept or identify CSAM manually. However, automatically detecting and analysing child sexual abusive language in online text is challenging and time-intensive, mostly due to the variety of data formats and privacy constraints of hosting platforms. We propose a CSAM detection intelligence algorithm based on natural language processing and machine learning techniques. Our CSAM detection model is not only used to remove CSAM on …

Exploring The Personality Of Virtual Tutors In Conversational Foreign Language Practice, Johanna Dobbriner, Cathy Ennis, Robert J. Ross

Conference papers

Fluid interaction between virtual agents and humans requires the understanding of many issues of conversational pragmatics. One such issue is the interaction between communication strategy and personality. As a step towards developing models of personality driven pragmatics policies, in this paper, we present our initial experiment to explore differences in user interaction with two contrasting avatar personalities. Each user saw a single personality in a video-call setting and gave feedback on the interaction. Our expectations, that a more extroverted outgoing positive personality would be a more successful tutor, were only partially confirmed. While this personality did induce longer conversations in …

Multi-Modal Self-Supervised Representation Learning For Earth Observation, Pallavi Jain, Bianca Schoen Phelan, Robert J. Ross

Conference papers

Self-Supervised learning (SSL) has reduced the performance gap between supervised and unsupervised learning, due to its ability to learn invariant representations. This is a boon to the domains like Earth Observation (EO), where labelled data availability is scarce but unlabelled data is freely available. While Transfer Learning from generic RGB pre-trained models is still common-place in EO, we argue that, it is essential to have good EO domain specific pre-trained model in order to use with downstream tasks with limited labelled data. Hence, we explored the applicability of SSL with multi-modal satellite imagery for downstream tasks. For this we utilised …

An Analysis Of The Interpretability Of Neural Networks Trained On Magnetic Resonance Imaging For Stroke Outcome Prediction, Esra Zihni, John D. Kelleher, Bryony Mcgarry

Conference papers

Applying deep learning models to MRI scans of acute stroke patients to extract features that are indicative of short-term outcome could assist a clinician’s treatment decisions. Deep learning models are usually accurate but are not easily interpretable. Here, we trained a convolutional neural network on ADC maps from hyperacute ischaemic stroke patients for prediction of short-term functional outcome and used an interpretability technique to highlight regions in the ADC maps that were most important in the prediction of a bad outcome. Although highly accurate, the model’s predictions were not based on aspects of the ADC maps related to stroke pathophysiology.

Wider Vision: Enriching Convolutional Neural Networks Via Alignment To External Knowledge Bases, Xuehao Liu, Sarah Jane Delany, Susan Mckeever

Conference papers

Deep learning models suffer from opaqueness. For Convolutional Neural Networks (CNNs), current research strategies for explaining models focus on the target classes within the associated training dataset. As a result, the understanding of hidden feature map activations is limited by the discriminative knowledge gleaned during training. The aim of our work is to explain and expand CNNs models via the mirroring or alignment of the network to an external knowledge base. This will allow us to give a semantic context or label for each visual feature. Using the resultant aligned embedding space, we can match CNN feature activations to nodes …

Zero-Shot Action Recognition With Knowledge Enhanced Generative Adversarial Networks, Kaiqiang Huang, Luis Miralles-Pechuán, Susan Mckeever

Conference papers

Zero-Shot Action Recognition (ZSAR) aims to recognise action classes in videos that have never been seen during model training. In some approaches, ZSAR has been achieved by generating visual features for unseen classes based on the semantic information of the unseen class labels using generative adversarial networks (GANs). Therefore, the problem is converted to standard supervised learning since the unseen visual features are accessible. This approach alleviates the lack of labelled samples of unseen classes. In addition, objects appearing in the action instances could be used to create enriched semantics of action classes and therefore, increase the accuracy of ZSAR. …

K-Nearest Neighbour Classifiers - A Tutorial, Padraig Cunningham, Sarah Jane Delany

Conference papers

Perhaps the most straightforward classifier in the arsenal or Machine Learning techniques is the Nearest Neighbour Classifier – classification is achieved by identifying the nearest neighbours to a query example and using those neighbours to determine the class of the query. This approach to classification is of particular importance because issues of poor run-time performance is not such a problem these days with the computational power that is available. This paper presents an overview of techniques for Nearest Neighbour classification focusing on; mechanisms for assessing similarity (distance), computational issues in identifying nearest neighbours and mechanisms for reducing the dimension of …

Virtual Tutor Personality In Computer Assisted Language Learning, Johanna Dobbriner, Cathy Ennis, Robert J. Ross

Conference papers

The use of intelligent virtual agents in language learning has increased in recent years. Studies into several aspects of personalisation aiming to increase user engagement are an ongoing research topic with avatar personality being one such aspect. As a step towards our development of intelligent virtual avatars, we present two of our initial experiments to explore differences in user interaction with two contrasting avatar personalities -- P1: open-minded, friendly and sociable and P2: closed-off, curt and distant. Each user interacted with a single personality in a video-call setting and gave feedback on the interaction. Our expectations, that P1 would be …

Language-Driven Region Pointer Advancement For Controllable Image Captioning, Annika Lindh, Robert J. Ross, John D. Kelleher

Conference papers

Controllable Image Captioning is a recent sub-field in the multi-modal task of Image Captioning wherein constraints are placed on which regions in an image should be described in the generated natural language caption. This puts a stronger focus on producing more detailed descriptions, and opens the door for more end-user control over results. A vital component of the Controllable Image Captioning architecture is the mechanism that decides the timing of attending to each region through the advancement of a region pointer. In this paper, we propose a novel method for predicting the timing of region pointer advancement by treating the …

Energy-Based Neural Modelling For Large-Scale Multiple Domain Dialogue State Tracking, Anh Duong Trinh, Robert J. Ross, John D. Kelleher

Conference papers

Scaling up dialogue state tracking to multiple domains is challenging due to the growth in the number of variables being tracked. Furthermore, dialog state tracking models do not yet explicitly make use of relationships between dialogue variables, such as slots across domains. We propose using energy-based structure prediction methods for large-scale dialogue state tracking task in two multiple domain dialogue datasets. Our results indicate that: (i) modelling variable dependencies yields better results; and (ii) the structured prediction output aligns with the dialogue slot-value constraint principles. This leads to promising directions to improve state-of-the-art models by incorporating variable dependencies into their …

F-Measure Optimisation And Label Regularisation For Energy-Based Neural Dialogue State Tracking Models, Anh Duong Trinh, Robert J. Ross, John D. Kelleher

Conference papers

In recent years many multi-label classification methods have exploited label dependencies to improve performance of classification tasks in various domains, hence casting the tasks to structured prediction problems. We argue that multi-label predictions do not always satisfy domain constraint restrictions. For example when the dialogue state tracking task in task-oriented dialogue domains is solved with multi-label classification approaches, slot-value constraint rules should be enforced following real conversation scenarios.

To address these issues we propose an energy-based neural model to solve the dialogue state tracking task as a structured prediction problem. Furthermore we propose two improvements over previous methods with respect …

Automatic Flood Detection In Sentinei-2 Images Using Deep Convolutional Neural Networks, Pallavi Jain, Bianca Schoen-Phelan, Robert J. Ross

Conference papers

The early and accurate detection of floods from satellite imagery can aid rescue planning and assessment of geophysical damage. Automatic identification of water from satellite images has historically relied on hand-crafted functions, but these often do not provide the accuracy and robustness needed for accurate and early flood detection. To try to overcome these limitations we investigate a tiered methodology combining water index like features with a deep convolutional neural network based solution to flood identification against the MediaEval 2019 flood dataset. Our method builds on existing deep neural network methods, and in particular the VGG16 network. Specifically, we explored …

Multimodal Fusion Strategies For Outcome Prediction In Stroke, Esra Zihni, John D. Kelleher, Vince I. Madai, Ahmed Khalil, Ivana Galinovic, Jochen Fiebach, Michelle Livne, Dietmar Frey

Conference papers

Data driven methods are increasingly being adopted in the medical domain for clinical predictive modeling. Prediction of stroke outcome using machine learning could provide a decision support system for physicians to assist them in patient-oriented diagnosis and treatment. While patient-specific clinical parameters play an important role in outcome prediction, a multimodal fusion approach that integrates neuroimaging with clinical data has the potential to improve accuracy. This paper addresses two research questions: (a) does multimodal fusion aid in the prediction of stroke outcome, and (b) what fusion strategy is more suitable for the task at hand. The baselines for our experimental …

Language Model Co-Occurrence Linking For Interleaved Activity Discovery, Eoin Rogers, Robert J. Ross, John D. Kelleher

Conference papers

As ubiquitous computer and sensor systems become abundant, the potential for automatic identification and tracking of human behaviours becomes all the more evident. Annotating complex human behaviour datasets to achieve ground truth for supervised training can however be extremely labour-intensive, and error prone. One possible solution to this problem is activity discovery: the identification of activities in an unlabelled dataset by means of an unsupervised algorithm. This paper presents a novel approach to activity discovery that utilises deep learning based language production models to construct a hierarchical, tree-like structure over a sequential vector of sensor events. Our approach differs from …

Modelling Interleaved Activities Using Language Models, Eoin Rogers, Robert J. Ross, John D. Kelleher

Conference papers

We propose a new approach to activity discovery, based on the neural language modelling of streaming sensor events. Our approach proceeds in multiple stages: we build binary links between activities using probability distributions generated by a neural language model trained on the dataset, and combine the binary links to produce complex activities. We then use the activities as sensor events, allowing us to build complex hierarchies of activities. We put an emphasis on dealing with interleaving, which represents a major challenge for many existing activity discovery systems. The system is tested on a realistic dataset, demonstrating it as a promising …

Mutual Information Decay Curves And Hyper-Parameter Grid Search Design For Recurrent Neural Architectures, Abhijit Mahalunkar, John Kelleher

Conference papers

We present an approach to design the grid searches for hyper-parameter optimization for recurrent neural architectures. The basis for this approach is the use of mutual information to analyze long distance dependencies (LDDs) within a dataset. We also report a set of experiments that demonstrate how using this approach, we obtain state-of-the-art results for DilatedRNNs across a range of benchmark datasets.

Update Frequency And Background Corpus Selection In Dynamic Tf-Idf Models For First Story Detection, Fei Wang, Robert J. Ross, John D. Kelleher

Conference papers

First Story Detection (FSD) requires a system to detect the very first story that mentions an event from a stream of stories. Nearest neighbour-based models, using the traditional term vector document representations like TF-IDF, currently achieve the state of the art in FSD. Because of its online nature, a dynamic term vector model that is incrementally updated during the detection process is usually adopted for FSD instead of a static model. However, very little research has investigated the selection of hyper-parameters and the background corpora for a dynamic model. In this paper, we analyse how a dynamic term vector model …

Capturing Dialogue State Variable Dependencies With An Energy-Based Neural Dialogue State Tracker, Anh Duong Trinh, Robert J. Ross, John D. Kelleher

Conference papers

Dialogue state tracking requires the population and maintenance of a multi-slot frame representation of the dialogue state. Frequently, dialogue state tracking systems assume independence between slot values within a frame. In this paper we argue that treating the prediction of each slot value as an independent prediction task may ignore important associations between the slot values, and, consequently, we argue that treating dialogue state tracking as a structured prediction problem can help to improve dialogue state tracking performance. To support this argument, the research presented in this paper is structured into three stages: (i) analyzing variable dependencies in dialogue data; …

Investigating Variable Dependencies In Dialogue States, Anh Duong Trinh, Robert J. Ross, John D. Kelleher

Conference papers

Dialogue State Tracking is arguably one of the most challenging tasks among dialogue processing problems due to the uncertainties of language and complexity of dialogue contexts. We argue that this problem is made more challenging by variable dependencies in the dialogue states that must be accounted for in processing. In this paper we give details on our motivation for this argument through statistical tests on a number of dialogue datasets. We also propose a machine learning-based approach called energy-based learning that tackles variable dependencies while performing prediction on the dialogue state tracking tasks.

Bigger Versus Similar: Selecting A Background Corpus For First Story Detection Based On Distributional Similarity, Fei Wang, Robert J. Ross, John D. Kelleher

Conference papers

The current state of the art for First Story Detection (FSD) are nearest neighbour-based models with traditional term vector representations; however, one challenge faced by FSD models is that the document representation is usually defined by the vocabulary and term frequency from a background corpus. Consequently, the ideal background corpus should arguably be both large-scale to ensure adequate term coverage, and similar to the target domain in terms of the language distribution. However, given these two factors cannot always be mutually satisfied, in this paper we examine whether the distributional similarity of common terms is more important than the scale …

Estimating Distributed Representation Performance In Disaster-Related Social Media Classification, Pallavi Jain, Robert J. Ross, Bianca Schoen-Phelan

Conference papers

This paper examines the effectiveness of a range of pre-trained language representations in order to determine the informativeness and information type of social media in the event of natural or man-made disasters. Within the context of disaster tweet analysis, we aim to accurately analyse tweets while minimising both false positive and false negatives in the automated information analysis. The investigation is performed across a number of well known disaster-related twitter datasets. Models that are built from pre-trained word embeddings from Word2Vec, GloVe, ELMo and BERT are used for performance evaluation. Given the relative ubiquity of BERT as a standout language …

Energy-Based Modelling For Dialogue State Tracking, Anh Duong Trinh, Robert J. Ross, John D. Kelleher

Conference papers

The uncertainties of language and the complexity of dialogue contexts make accurate dialogue state tracking one of the more challenging aspects of dialogue processing. To improve state tracking quality, we argue that relationships between different aspects of dialogue state must be taken into account as they can often guide a more accurate interpretation process. To this end, we present an energy-based approach to dialogue state tracking as a structured classification task. The novelty of our approach lies in the use of an energy network on top of a deep learning architecture to explore more signal correlations between network variables including …