Multimodal Fusion For Audio-Image And Video Action Recognition, Muhammad B. Shaikh, Douglas Chai, Syed M. S. Islam, Naveed Akhtar

Research outputs 2022 to 2026

Multimodal Human Action Recognition (MHAR) is an important research topic in computer vision and event recognition fields. In this work, we address the problem of MHAR by developing a novel audio-image and video fusion-based deep learning framework that we call Multimodal Audio-Image and Video Action Recognizer (MAiVAR). We extract temporal information using image representations of audio signals and spatial information from video modality with the help of Convolutional Neutral Networks (CNN)-based feature extractors and fuse these features to recognize respective action classes. We apply a high-level weights assignment algorithm for improving audio-visual interaction and convergence. This proposed fusion-based framework utilizes …

Sc-Fuse: A Feature Fusion Approach For Unpaved Road Detection From Remotely Sensed Images, Aniruddh Saxena

Department of Computer Science and Engineering: Dissertations, Theses, and Student Research

Road network extraction from remote sensing imagery is crucial for numerous applications, ranging from autonomous navigation to urban and rural planning. A particularly challenging aspect is the detection of unpaved roads, often underrepresented in research and data. These roads display variability in texture, width, shape, and surroundings, making their detection quite complex. This thesis addresses these challenges by creating a specialized dataset and introducing the SC-Fuse model.

Our custom dataset comprises high resolution remote sensing imagery which primarily targets unpaved roads of the American Midwest. To capture the diverse seasonal variation and their impact, the dataset includes images from different …

An In-Depth Analysis Of Domain Adaptation In Computer And Robotic Vision, Muhammad Hassan Tanveer, Zainab Fatima, Shehnila Zardari, David A. Guerra-Zubiaga

Faculty and Research Publications

This review article comprehensively delves into the rapidly evolving field of domain adaptation in computer and robotic vision. It offers a detailed technical analysis of the opportunities and challenges associated with this topic. Domain adaptation methods play a pivotal role in facilitating seamless knowledge transfer and enhancing the generalization capabilities of computer and robotic vision systems. Our methodology involves systematic data collection and preparation, followed by the application of diverse assessment metrics to evaluate the efficacy of domain adaptation strategies. This study assesses the effectiveness and versatility of conventional, deep learning-based, and hybrid domain adaptation techniques within the domains of …

An Empirical Study Of Pre-Trained Model Reuse In The Hugging Face Deep Learning Model Registry, Wenxin Jiang, Nicholas Synovic, Matt Hyatt, Taylor R. Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K. Thiruvathukal, James C. Davis

Department of Electrical and Computer Engineering Faculty Publications

Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems.

In this work, we present the first empirical investigation of PTM reuse. …

Security Of Internet Of Things (Iot) Using Federated Learning And Deep Learning — Recent Advancements, Issues And Prospects, Vinay Gugueoth, Sunitha Safavat, Sachin Shetty

Electrical & Computer Engineering Faculty Publications

There is a great demand for an efficient security framework which can secure IoT systems from potential adversarial attacks. However, it is challenging to design a suitable security model for IoT considering the dynamic and distributed nature of IoT. This motivates the researchers to focus more on investigating the role of machine learning (ML) in the designing of security models. A brief analysis of different ML algorithms for IoT security is discussed along with the advantages and limitations of ML algorithms. Existing studies state that ML algorithms suffer from the problem of high computational overhead and risk of privacy leakage. …

Gated Deep Reinforcement Learning With Red Deer Optimization For Medical Image Classification, Narayanan Ganesh, Sambandan Jayalakshmi, Rama Chandran Narayanan, Miroslav Mahdal, Hossam Zawbaa, Ali Wagdy Mohamed

Articles

The brain is one of the most important and complex organs in the body, consisting of billions of individual cells. Uncontrolled growth and expansion of aberrant cell populations within or around the brain are the main causes of brain tumors. These cells have the potential to harm healthy cells and impair brain function [1]. Tumors can be detected using medical imaging techniques, which are considered the most popular and accurate way to classify different types of cancer, and this procedure is even more crucial as it is noninvasive [2]. Magnetic resonance imaging (MRI) is one such medical imaging technique that …

How Visual Stimuli Evoked P300 Is Transforming The Brain–Computer Interface Landscape: A Prisma Compliant Systematic Review, Jai Kalra, Prashasti Mittal, Nirmiti Mittal, Abhishek Arora, Utkarsh Tewari, Aviral Chharia, Rahul Upadhyay, Vinay Kumar, Luca Longo

Articles

Non-invasive Visual Stimuli evoked-EEGbased P300 BCIs have gained immense attention in recent years due to their ability to help patients with disability using BCI-controlled assistive devices and applications. In addition to the medical field, P300 BCI has applications in entertainment, robotics, and education. The current article systematically reviews 147 articles that were published between 2006-2021*. Articles that pass the pre-defined criteria are included in the study. Further, classification based on their primary focus, including article orientation, participants’ age groups, tasks given, databases, the EEG devices used in the studies, classification models, and application domain, is performed. The application-based classification considers …

Schizo-Net: A Novel Schizophrenia Diagnosis Framework Using Late Fusion Multimodal Deep Learning On Electroencephalogram-Based Brain Connectivity Indices, Nitin Grover, Aviral Chharia, Rahul Upadhyay, Luca Longo

Articles

Schizophrenia (SCZ) is a serious mental condition that causes hallucinations, delusions, and disordered thinking. Traditionally, SCZ diagnosis involves the subject’s interview by a skilled psychiatrist. The process needs time and is bound to human errors and bias. Recently, brain connectivity indices have been used in a few pattern recognition methods to discriminate neuro-psychiatric patients from healthy subjects. The study presents Schizo-Net , a novel, highly accurate, and reliable SCZ diagnosis model based on a late multimodal fusion of estimated brain connectivity indices from EEG activity. First, the raw EEG activity is pre-processed exhaustively to remove unwanted artifacts. Next, six brain …

Effective Short Text Classification Via The Fusion Of Hybrid Features For Iot Social Data, Xiong Luo, Zhijian Yu, Zhigang Zhao, Wenbing Zhao, Jenq-Haur Wang

Electrical and Computer Engineering Faculty Publications

Nowadays short texts can be widely found in various social data in relation to the 5G-enabled Internet of Things (IoT). Short text classification is a challenging task due to its sparsity and the lack of context. Previous studies mainly tackle these problems by enhancing the semantic information or the statistical information individually. However, the improvement achieved by a single type of information is limited, while fusing various information may help to improve the classification accuracy more effectively. To fuse various information for short text classification, this article proposes a feature fusion method that integrates the statistical feature and the comprehensive …

Generating Reality-Analogous Datasets For Autonomous Uav Navigation Using Digital Twin Areas, Thomas Lee, Susan Mckeever, Jane Courtney

Conference papers

In order for autonomously navigating Unmanned Air Vehicles(UAVs) to be implemented in day-to-day life, proof of safe operation will be necessary for all realistic navigation scenarios. For Deep Learning powered navigation protocols, this requirement is challenging to fulfil as the performance of a network is impacted by how much the test case deviates from data that the network was trained on. Though networks can generalise to manage multiple scenarios in the same task, they require additional data representing those cases which can be costly to gather. In this work, a solution to this data acquisition problem is suggested by way …

Estimating Animal Pose Using Deep Learning A Trained Deep Learning Model Outperforms Morphological Analysis, Sanghoon Lee, Jarod Banzon, Kevin Le, Dal Hyung Kim

Computer Sciences and Electrical Engineering Faculty Research

INTRODUCTION: Analyzing animal behavior helps researchers understand their decision-making process and helper tools are rapidly becoming an indispensable part of many interdisciplinary studies. However, researchers are often challenged to estimate animal pose because of the limitation of the tools and its vulnerability to a specific environment. Over the years, deep learning has been introduced as an alternative solution to overcome these challenges.

OBJECTIVES: This study investigates how deep learning models can be applied for the accurate prediction of animal behavior, comparing with traditional morphological analysis based on image pixels.

METHODS: Transparent Omnidirectional Locomotion Compensator (TOLC), a tracking device, is used …

A Deep Learning-Based Approach To Extraction Of Filler Morphology In Sem Images With The Application Of Automated Quality Inspection, Md. Fashiar Rahman, Tzu-Liang Bill Tseng, Jianguo Wu, Yuxin Wen, Yirong Lin

Engineering Faculty Articles and Research

Automatic extraction of filler morphology (size, orientation, and spatial distribution) in Scanning Electron Microscopic (SEM) images is essential in many applications such as automatic quality inspection in composite manufacturing. Extraction of filler morphology greatly depends on accurate segmentation of fillers (fibers and particles), which is a challenging task due to the overlap of fibers and particles and their obscure presence in SEM images. Convolution Neural Networks (CNNs) have been shown to be very effective at object recognition in digital images. This paper proposes an automatic filler detection system in SEM images, utilizing a Mask Region-based CNN architecture. The proposed system …

Ensemble Approach To The Semantic Segmentation Of Satellite Images, Brendan Kent

Dissertations

Automatic classification and segmentation of land use land cover(LULC) is extremely important for understanding the relationship between humans and nature. Human pressures on the environment have drastically accelerated in the last decades, risking biodiversity and ecosystem services. Remote sensing via satellite imagery is an excellent tool to study LULC. Research has shown that deep learning encoder-decoder architectures have achieved worthy results in the area of LULC, however the application of an ensemble approach has not been well quantified. Studies have shown it to be useful in the area of medical imaging. Ensembling by pooling together predictions to produce better predictions …

Evaluating The Performance Of Vision Transformer Architecture For Deepfake Image Classification, Devesan Govindasamy

Dissertations

Deepfake classification has seen some impressive results lately, with the experimentation of various deep learning methodologies, researchers were able to design some state-of-the art techniques. This study attempts to use an existing technology “Transformers” in the field of Natural Language Processing (NLP) which has been a de-facto standard in text processing for the purposes of Computer Vision. Transformers use a mechanism called “self-attention”, which is different from CNN and LSTM. This study uses a novel technique that considers images as 16x16 words (Dosovitskiy et al., 2021) to train a deep neural network with “self-attention” blocks to detect deepfakes. It creates …

Evaluation Of Automated Eye Blink Artefact Removal Using Stacked Dense Autoencoder, Matthew Rigney

Dissertations

The presence of artefacts in Electroencephalograph (EEG) signals can have a considerable impact on the information they portray. In this comparative study, the automated removal of eye blink artefacts using the constrained latent representation of a stacked dense autoencoders (SDAE) and comparing its ability to that of the manual independent component analysis (ICA) approach was evaluated. A comparative evaluation of 5 stacked dense autoencoder architectures lead to a chosen architecture for which the ability to automatically detect and remove eye blink artefacts were both statistically and humanistically evaluated. The ability of the stacked dense autoencoder was statistically evaluated with the …

Bfv-Based Homomorphic Encryption For Privacy-Preserving Cnn Models, Febrianti Wibawa, Ferhat Ozgur Catak, Salih Sarp, Murat Kuzlu

Engineering Technology Faculty Publications

Medical data is frequently quite sensitive in terms of data privacy and security. Federated learning has been used to increase the privacy and security of medical data, which is a sort of machine learning technique. The training data is disseminated across numerous machines in federated learning, and the learning process is collaborative. There are numerous privacy attacks on deep learning (DL) models that attackers can use to obtain sensitive information. As a result, the DL model should be safeguarded from adversarial attacks, particularly in medical data applications. Homomorphic encryption-based model security from the adversarial collaborator is one of the answers …

Modeling Cognitive Load As A Self-Supervised Brain Rate With Electroencephalography And Deep Learning, Luca Longo

Articles

The principal reason for measuring mental workload is to quantify the cognitive cost of performing tasks to predict human performance. Unfortunately, a method for assessing mental workload that has general applicability does not exist yet. This is due to the abundance of intuitions and several operational definitions from various fields that disagree about the sources or workload, its attributes, the mechanisms to aggregate these into a general model and their impact on human performance. This research built upon these issues and presents a novel method for mental workload modelling from EEG data employing deep learning. This method is self-supervised, employing …

Breast Cancer Detection From Histopathology Images Using Machine Learning Techniques: A Bibliometric Analysis, Shubhangi A. Joshi, Anupkumar M. Bongale Dr., Arunkumar M. Bongale Dr.

Library Philosophy and Practice (e-journal)

Computer aided diagnosis has become upcoming area of research over past few years. With the advent of machine learning and especially deep learning techniques, the scenario of work flow management in healthcare sector is changing drastically. Artificial intelligence has shown potential in the field of breast cancer care. With datasets for machine learning frameworks getting eventually richer with time, we can definitely get newer insights in the field of breast cancer care. This will help in narrowing down the treatment range for patients and increasing patient survivability. The purpose of this study was to perform bibliometric analysis of the literature …

Artificial Intelligence And The Ethics Behind It, Isaac Johnston

Senior Honors Theses

Artificial intelligence (AI) has been a widely used buzzword for the past couple of years. If there is a technology that works without human interaction, it is labeled as AI. But what is AI, and should individuals be concerned? The following research aims to define what artificial intelligence is, specifically machine learning (ML) and neural networks. It is important to understand how AI is used today in cars, image recognition, ad marketing, and other areas. Although AI has many benefits, there are areas of ethical concerns such as autonomous cars, military applications, social media marketing, and others. This paper helps …

Quantitative Analysis Of Research On Artificial Intelligence In Retinopathy Of Prematurity, Ranjana Agrawal, Manasi Anup Agrawal, Sucheta Kulkarni, Ketan Kotecha, Rahee Walambe

Library Philosophy and Practice (e-journal)

Retinopathy of Prematurity (ROP) is a disease of the eye and a potential source of blindness in low birth weight preterm infants. It is preventable if diagnosed and treated on time. Artificial Intelligence (AI) has played an important role in developing automated screening systems to assist medical experts. There are many traditional literature review articles available that focus on the scientific content of ROP-AI. The researchers also require a bibliometric analysis to become acquainted with the competing groups and new trends in this field. This paper gives a brief overview of ROP and AI systems for ROP screening with a …

On-Device Deep Learning Inference For System-On-Chip (Soc) Architectures, Tom Springer, Elia Eiroa-Lledo, Elizabeth Stevens, Erik Linstead

Engineering Faculty Articles and Research

As machine learning becomes ubiquitous, the need to deploy models on real-time, embedded systems will become increasingly critical. This is especially true for deep learning solutions, whose large models pose interesting challenges for target architectures at the “edge” that are resource-constrained. The realization of machine learning, and deep learning, is being driven by the availability of specialized hardware, such as system-on-chip solutions, which provide some alleviation of constraints. Equally important, however, are the operating systems that run on this hardware, and specifically the ability to leverage commercial real-time operating systems which, unlike general purpose operating systems such as Linux, can …

Wg2An: Synthetic Wound Image Generation Using Generative Adversarial Network, Salih Sarp, Murat Kuzlu, Emmanuel Wilson, Ozgur Guler

Engineering Technology Faculty Publications

In part due to its ability to mimic any data distribution, Generative Adversarial Network (GAN) algorithms have been successfully applied to many applications, such as data augmentation, text-to-image translation, image-to-image translation, and image inpainting. Learning from data without crafting loss functions for each application provides broader applicability of the GAN algorithm. Medical image synthesis is also another field that the GAN algorithm has great potential to assist clinician training. This paper proposes a synthetic wound image generation model based on GAN architecture to increase the quality of clinical training. The proposed model is trained on chronic wound datasets with various …

Bibliometric Analysis Of One-Stage And Two-Stage Object Detection, Aditya Lohia, Kalyani Dhananjay Kadam, Rahul Raghvendra Joshi, Dr. Anupkumar M. Bongale

Library Philosophy and Practice (e-journal)

Object Detection using deep learning has seen a boom in the recent couple of years. Observing the trend and its research, it is important to summarize bibliometrics related to object detection which will help researchers contribute to this subject area. This paper details bibliometrics for one-stage object detection and two-stage object detection. This uses Scopus database for data analysis. This also uses tools like Sciencescape, Gephi, etc. It can be observed that the advancements to the field of object detection are seen in recent years and explored to its full extent. It is observed that Chinese universities and researchers are …

A Bibliometric Analysis Of Plant Disease Classification With Artificial Intelligence Based On Scopus And Wos, Shivali Amit Wagle, Harikrishnan R

Library Philosophy and Practice (e-journal)

The maneuver of Artificial Intelligence (AI) techniques in the field of agriculture help in the classification of diseases. Early prediction of the disease benefits in taking relevant management steps. This is an important step towards controlling the disease growth that will yield good quality products to fulfill the global food demand. The main objective of this paper is to study the extent of research work done in this area of plant disease classification. The paper discusses the bibliometric analysis of plant disease classification with AI in Scopus and Web of Science core collection (WOS) database in analyzing the research by …

A Systematic Review Of Convolutional Neural Network-Based Structural Condition Assessment Techniques, Sandeep Sony, Kyle Dunphy, Ayan Sadhu, Miriam A M Capretz

Electrical and Computer Engineering Publications

With recent advances in non-contact sensing technology such as cameras, unmanned aerial and ground vehicles, the structural health monitoring (SHM) community has witnessed a prominent growth in deep learning-based condition assessment techniques of structural systems. These deep learning methods rely primarily on convolutional neural networks (CNNs). The CNN networks are trained using a large number of datasets for various types of damage and anomaly detection and post-disaster reconnaissance. The trained networks are then utilized to analyze newer data to detect the type and severity of the damage, enhancing the capabilities of non-contact sensors in developing autonomous SHM systems. In recent …

Demystifying Artificial Intelligence Based Behavior Prediction Of Traffic Actors For Autonomous Vehicle- A Bibliometric Analysis Of Trends And Techniques, Suresh Sudam Kolekar, Shilpa Shailesh Gite, Biswajeet Pradhan

Library Philosophy and Practice (e-journal)

Background: The purpose of this study is to examine, using bibliometric methods, the work done on behavior prediction of traffic actors for autonomous vehicles using various artificial intelligence algorithms from 2011 to 2020.

Methods: Using one of the most common databases, Scopus, numerous papers on behavior prediction of traffic actors for autonomous vehicles were retrieved. The research papers are being considered for the period from 2011 to 2020. The Scopus analyzer is used to obtain some results of the study, such as documents by year, source, and country and so on. VOSviewer Version 1.6.16 is used for the analysis of …

Converting Optical Videos To Infrared Videos Using Attention Gan And Its Impact On Target Detection And Classification Performance, Mohammad Shahab Uddin, Reshad Hoque, Kazi Aminul Islam, Chiman Kwan, David Gribben, Jiang Li

Electrical & Computer Engineering Faculty Publications

To apply powerful deep-learning-based algorithms for object detection and classification in infrared videos, it is necessary to have more training data in order to build high-performance models. However, in many surveillance applications, one can have a lot more optical videos than infrared videos. This lack of IR video datasets can be mitigated if optical-to-infrared video conversion is possible. In this paper, we present a new approach for converting optical videos to infrared videos using deep learning. The basic idea is to focus on target areas using attention generative adversarial network (attention GAN), which will preserve the fidelity of target areas. …

Adequately Generating Captions For An Image Using Adaptive And Global Attention Mechanisms., Shravan Kumar Talanki Venkatarathanaiahsetty

Dissertations

Generating description to images is a recent surge and with latest developments in the field of Artificial Intelligence, it can be one of the prominent applications to bridge the gap between Computer vision and Natural language processing fields. In terms of the learning curve, Deep learning has become the main backbone in driving many new applications. Image Captioning is one such application where the usage of Deep learning methods enhanced the performance of the captioning accuracy. The introduction of the Encoder-Decoder framework was a breakthrough in Image captioning. But as the sequences got longer the performance of captions was affected. …

Evaluating The Performance Of Transformer Architecture Over Attention Architecture On Image Captioning, Deepti Balasubramaniam

Dissertations

Over the last few decades computer vision and Natural Language processing has shown tremendous improvement in different tasks such as image captioning, video captioning, machine translation etc using deep learning models. However, there were not much researches related to image captioning based on transformers and how it outperforms other models that were implemented for image captioning. In this study will be designing a simple encoder-decoder model, attention model and transformer model for image captioning using Flickr8K dataset where will be discussing about the hyperparameters of the model, type of pre-trained model used and how long the model has been trained. …

Bibliometric Review On Image Based Plant Phenotyping, Shrikrishna Ulhas Kolhar, Jayant Jagtap

Library Philosophy and Practice (e-journal)

Plant phenotyping is a quantitative description of structural, physiological and temporal traits of plants resulting from interaction of plant genotypes with the environment. A rapid development is in progress in the field of image-based plant phenotyping. Plant phenotyping has wide range of applications in plant breeding research, plant growth prediction, biotic and abiotic stress analysis, crop management and early disease detection. The main motive is to provide detailed bibliometric review in order to know the available literature and current research trends in the area of plant phenotyping using plant images. The bibliometric analysis is primarily based on Scopus, web of …