Physical Sciences and Mathematics Commons^™

Intelligent Solutions For Retroactive Anomaly Detection And Resolution With Log File Systems, Derek G. Rogers, Chanvo Nguyen, Abhay Sharma

SMU Data Science Review

This paper explores the intricate challenges log files pose from data science and machine learning perspectives. Drawing inspiration from existing methods, LAnoBERT, PULL, LLMs, and the breadth of recent research, this paper aims to push the boundaries of machine learning for log file systems. Our study comprehensively examines the unique challenges presented in our problem setup, delineates the limitations of existing methods, and introduces innovative solutions. These contributions are organized to offer valuable insights, predictions, and actionable recommendations tailored for Microsoft's engineers working on log data analysis.

Achieving Responsible Anomaly Detection, Xiao Han

All Graduate Theses and Dissertations, Fall 2023 to Present

In the digital transformation era, safeguarding online systems against anomalies – unusual patterns indicating potential threats or malfunctions – has become crucial. This dissertation embarks on enhancing the accuracy, explainability, and ethical integrity of anomaly detection systems. By integrating advanced machine learning techniques, it improves anomaly detection performance and incorporates fairness and explainability at its core.

The research tackles performance enhancement in anomaly detection by leveraging few-shot learning, demonstrating how systems can effectively identify anomalies with minimal training data. This approach overcomes data scarcity challenges. Reinforcement learning is employed to iteratively refine models, enhancing decision-making processes. Transfer learning enables the …

Railroad Condition Monitoring Using Distributed Acoustic Sensing And Deep Learning Techniques, Md Arifur Rahman

Electronic Theses and Dissertations

Proper condition monitoring has been a major issue among railroad administrations since it might cause catastrophic dilemmas that lead to fatalities or damage to the infrastructure. Although various aspects of train safety have been conducted by scholars, in-motion monitoring detection of defect occurrence, cause, and severity is still a big concern. Hence extensive studies are still required to enhance the accuracy of inspection methods for railroad condition monitoring (CM). Distributed acoustic sensing (DAS) has been recognized as a promising method because of its sensing capabilities over long distances and for massive structures. As DAS produces large datasets, algorithms for precise …

Weakly-Supervised Anomaly Detection In Surveillance Videos Based On Two-Stream I3d Convolution Network, Sareh Soltani Nejad

Electronic Thesis and Dissertation Repository

The widespread adoption of city surveillance systems has led to an increase in the use of surveillance videos for maintaining public safety and security. This thesis tackles the problem of detecting anomalous events in surveillance videos. The goal is to automatically identify abnormal events by learning from both normal and abnormal videos. Most of previous works consider any deviation from learned normal patterns as an anomaly, which may not always be valid since the same activity could be normal or abnormal under different circumstances. To address this issue, the thesis utilizes the Two-Stream Inflated 3D (I3D) Convolutional Networks to extract …

Deep Isolation Forest For Anomaly Detection, Hongzuo Xu, Guansong Pang, Yijie Wang, Yongjun Wang

Research Collection School Of Computing and Information Systems

Isolation forest (iForest) has been emerging as arguably the most popular anomaly detector in recent years due to its general effectiveness across different benchmarks and strong scalability. Nevertheless, its linear axis-parallel isolation method often leads to (i) failure in detecting hard anomalies that are difficult to isolate in high-dimensional/non-linear-separable data space, and (ii) notorious algorithmic bias that assigns unexpectedly lower anomaly scores to artefact regions. These issues contribute to high false negative errors. Several iForest extensions are introduced, but they essentially still employ shallow, linear data partition, restricting their power in isolating true anomalies. Therefore, this paper proposes deep isolation …

Real–Time Semantic Segmentation For Railway Anomalies Analysis, Paul Stanik Iii

UNLV Theses, Dissertations, Professional Papers, and Capstones

In the past few years, computer vision has made huge jumps due to deep learning which leverages increased computational power and access to data. The computer vision community has also embraced transparency to accelerate research progress by sharing open datasets and open source code. Access to large scale datasets and benchmark challenges propelled and opened the field. The autonomous vehicle community is a prime example. While there has been significant growth in the automotive vision community, not much has been done in the rail domain. Traditional rail inspection methods require special trains that are run during down time, have sensitive …

Anomaly Detection In Sequential Data: A Deep Learning-Based Approach, Jayesh Soni

FIU Electronic Theses and Dissertations

Anomaly Detection has been researched in various domains with several applications in intrusion detection, fraud detection, system health management, and bio-informatics. Conventional anomaly detection methods analyze each data instance independently (univariate or multivariate) and ignore the sequential characteristics of the data. Anomalies in the data can be detected by grouping the individual data instances into sequential data and hence conventional way of analyzing independent data instances cannot detect anomalies. Currently: (1) Deep learning-based algorithms are widely used for anomaly detection purposes. However, significant computational overhead time is incurred during the training process due to static constant batch size and learning …

Anomaly Detection Methods To Improve Supply Chain Data Quality And Operations, Ana E. Glaser, Jake P. Harrison, David Josephs

SMU Data Science Review

Supply chain operations drive the planning, manufacture, and distribution of billions of semiconductors a year, spanning thousands of products across many supply chain configurations. The customizations span from wafer technology to die stacking and chip feature enablement. Data quality drives efficiency in these processes and anomalies in data can be very disruptive, and at times, consequential. Developing preventative measures that automate the detection of anomalies before they reach downstream execution systems would result in significant efficiency gain for the organization. The purpose of this research is to identify an effective, actionable, and computationally efficient approach to highlight anomalies in a …

Hybrid Machine And Deep Learning-Based Cyberattack Detection And Classification In Smart Grid Networks, Adedayo Aribisala

Electronic Theses and Dissertations

Power grids have rapidly evolved into Smart grids and are heavily dependent on Supervisory Control and Data Acquisition (SCADA) systems for monitoring and control. However, this evolution increases the susceptibility of the remote (VMs, VPNs) and physical interfaces (sensors, PMUs LAN, WAN, sub-stations power lines, and smart meters) to sophisticated cyberattacks. The continuous supply of power is critical to power generation plants, power grids, industrial grids, and nuclear grids; the halt to global power could have a devastating effect on the economy's critical infrastructures and human life.

Machine Learning and Deep Learning-based cyberattack detection modeling have yielded promising results when …

Noise Resilient Learning For Attack Detection In Smart Grid Pmu Infrastructure, Prithwiraj Roy, Shameek Bhattacharjee, Sahar Abedzadeh, Sajal K. Das

Computer Science Faculty Research & Creative Works

Falsified data from compromised Phasor Measurement Units (PMUs) in a smart grid induce Energy Management Systems (EMS) to have an inaccurate estimation of the state of the grid, disrupting various operations of the power grid. Moreover, the PMUs deployed at the distribution layer of a smart grid show dynamic fluctuations in their data streams, which make it extremely challenging to design effective learning frameworks for anomaly-based attack detection. In this paper, we propose a noise resilient learning framework for anomaly-based attack detection specifically for distribution layer PMU infrastructure, that show real time indicators of data falsifications attacks while offsetting the …

Active Learning Augmented Folded Gaussian Model For Anomaly Detection In Smart Transportation, Venkata Praveen Kumar Madhavarapu, Prithwiraj Roy, Shameek Bhattacharjee, Sajal K. Das

Computer Science Faculty Research & Creative Works

Smart transportation networks have become instrumental in smart city applications with the potential to enhance road safety, improve the traffic management system and driving experience. A Traffic Message Channel (TMC) is an IoT device that records the data collected from the vehicles and forwards it to the Roadside Units (RSUs). This data is further processed and shared with the vehicles to inquire the fastest route and incidents that can cause significant delays. The failure of the TMC sensors can have adverse effects on the transportation network. In this paper, we propose a Gaussian distribution-based trust scoring model to identify anomalous …

Anomaly Based Incident Detection In Large Scale Smart Transportation Systems, Jaminur Islam, Jose Paolo Talusan, Shameek Bhattacharjee, Francis Tiausas, Sayyed Mohsen Vazirizade, Abhishek Dubey, Keiichi Yasumoto, Sajal K. Das

Computer Science Faculty Research & Creative Works

Modern smart cities are focusing on smart transportation solutions to detect and mitigate the effects of various traffic incidents in the city. To materialize this, roadside units and ambient trans-portation sensors are being deployed to collect vehicular data that provides real-time traffic monitoring. In this paper, we first propose a real-time data-driven anomaly-based traffic incident detection framework for a city-scale smart transportation system. Specifically, we propose an incremental region growing approximation algorithm for optimal Spatio-temporal clustering of road segments and their data; such that road segments are strategically divided into highly correlated clusters. The highly correlated clusters enable identifying a …

Detecting Malicious Dns Queries Over Encrypted Tunnels Using Statistical Analysis And Bi-Directional Recurrent Neural Networks, Mohammad Al-Fawa'reh, Zain Ashi, Mousa Tayseer Jafar

Karbala International Journal of Modern Science

The exponential rise in the number of malicious threats targeting computer networks and digital services puts network infrastructure in jeopardy. Domain name protocol attacks are one of the most pervasive network attacks posing a threat to networks, whereby attackers send harmful information to the network; this type of threat is identified as DNS tunneling. The DNS protocol has recently gained increased attention from cyber-attackers, targeting organizations with a web presence or reliance on e-commerce businesses. Cyber-attackers can subtly exploit the contents of encrypted DNS packets that are sent across covert network tunnels, which are difficult for firewalls and blacklist detection …

Advanced Analytics In Smart Manufacturing: Anomaly Detection Using Machine Learning Algorithms And Parallel Machine Scheduling Using A Genetic Algorithm, Meiling He

Theses and Dissertations

Industry 4.0 offers great opportunities to utilize advanced data processing tools by generating Big Data from a more connected and efficient data collection system. Making good use of data processing technologies, such as machine learning and optimization algorithms, will significantly contribute to better quality control, automation, and job scheduling in Smart Manufacturing. This research aims to develop a new machine learning algorithm for solving highly imbalanced data processing problems, implement both supervised and unsupervised machine learning auto-selection frameworks for detecting anomalies in smart manufacturing, and develop a genetic algorithm for optimizing job schedules on unrelated parallel machines. This research also …

Toward Deep Supervised Anomaly Detection: Reinforcement Learning From Partially Labeled Anomaly Data, Guansong Pang, Anton Van Den Hengel, Chunhua Shen, Longbing Cao

Research Collection School Of Computing and Information Systems

We consider the problem of anomaly detection with a small set of partially labeled anomaly examples and a large-scale unlabeled dataset. This is a common scenario in many important applications. Existing related methods either exclusively fit the limited anomaly examples that typically do not span the entire set of anomalies, or proceed with unsupervised learning from the unlabeled data. We propose here instead a deep reinforcement learning-based approach that enables an end-to-end optimization of the detection of both labeled and unlabeled anomalies. This approach learns the known abnormality by automatically interacting with an anomalybiased simulation environment, while continuously extending the …

Semi-Supervised Spatial-Temporal Feature Learning On Anomaly-Based Network Intrusion Detection, Huy Mai

Computer Science and Computer Engineering Undergraduate Honors Theses

Due to a rapid increase in network traffic, it is growing more imperative to have systems that detect attacks that are both known and unknown to networks. Anomaly-based detection methods utilize deep learning techniques, including semi-supervised learning, in order to effectively detect these attacks. Semi-supervision is advantageous as it doesn't fully depend on the labelling of network traffic data points, which may be a daunting task especially considering the amount of traffic data collected. Even though deep learning models such as the convolutional neural network have been integrated into a number of proposed network intrusion detection systems in recent years, …

Automl For Anomaly Detection Of Time Series And Sequences Of Short Text, Cynthia Freeman

Computer Science ETDs

Automated approaches for parameter and algorithm selection greatly democratize fields such as machine learning, saving time and money as hiring experts can be prohibitively expensive. Unfortunately, anomaly detection is difficult to automate due to subjectivity and class imbalance. An anomaly detection system is presented that incorporates human-in-the-loop techniques and is dynamic, scalable, and able to work with non-annotated data. By focusing on meta-features of the input data, the system can intelligently choose the most promising anomaly detection methods. The system is agnostic to the medium of data; it only expects the data to be sequential in nature.

Establishing Behavioral Baselines For Computational Systems: Two Case Studies, John Henry Ring

Graduate College Dissertations and Theses

The behavior of modern systems lives in a complex landscape that is unique to its particular application. In this work we describe and analyze the behavior of two modern computational systems: a Linux server and the National Market System (NMS). Though this work is diverse in both the type and scale of system under study, it is unified through the design and implementation of computationally tractable quantitative metrics aimed at defining the state of behavior of these systems. Understanding the behavior of these systems allows us to ensure their desired operation. In the case of a server we need to …

Automatic Fall Risk Detection Based On Imbalanced Data, Yen-Hung Liu, Patrick C. K. Hung, Farkhund Iqbal, Benjamin C. M. Fung

All Works

In recent years, the declining birthrate and aging population have gradually brought countries into an ageing society. Regarding accidents that occur amongst the elderly, falls are an essential problem that quickly causes indirect physical loss. In this paper, we propose a pose estimation-based fall detection algorithm to detect fall risks. We use body ratio, acceleration and deflection as key features instead of using the body keypoints coordinates. Since fall data is rare in real-world situations, we train and evaluate our approach in a highly imbalanced data setting. We assess not only different imbalanced data handling methods but also different machine …

Automated Anomaly Detection And Localization System For A Microservices Based Cloud System, Priyanka Prakash Naikade

Electronic Thesis and Dissertation Repository

Context: With an increasing number of applications running on a microservices-based cloud system (such as AWS, GCP, IBM Cloud), it is challenging for the cloud providers to offer uninterrupted services with guaranteed Quality of Service (QoS) factors. Problem Statement: Existing monitoring frameworks often do not detect critical defects among a large volume of issues generated, thus affecting recovery response times and usage of maintenance human resource. Also, manually tracing the root causes of the issues requires a significant amount of time. Objective: The objective of this work is to: (i) detect performance anomalies, in real-time, through monitoring KPIs (Key Performance …

Next-Generation Self-Organizing Communications Networks: Synergistic Application Of Machine Learning And User-Centric Technologies, Chetana V. Murudkar

USF Tampa Graduate Theses and Dissertations

The telecommunications industry is going through a metamorphic journey where the 5G and 6G technologies will be deeply rooted in the society forever altering how people access and use information. In support of this transformation, this dissertation proposes a fundamental paradigm shift in the design, performance assessment, and optimization of wireless communications networks developing the next-generation self-organizing communications networks with the synergistic application of machine learning and user-centric technologies.

This dissertation gives an overview of the concept of self-organizing networks (SONs), provides insight into the “hot” technology of machine learning (ML), and offers an intuitive understanding of the user-centric (UC) …

Representation Learning With Adversarial Latent Autoencoders, Stanislav Pidhorskyi M.S.

Graduate Theses, Dissertations, and Problem Reports

A large number of deep learning methods applied to computer vision problems require encoder-decoder maps. These methods include, but are not limited to, self-representation learning, generalization, few-shot learning, and novelty detection. Encoder-decoder maps are also useful for photo manipulation, photo editing, superresolution, etc. Encoder-decoder maps are typically learned using autoencoder networks.
Traditionally, autoencoder reciprocity is achieved in the image-space using pixel-wise
similarity loss, which has a widely known flaw of producing non-realistic reconstructions. This flaw is typical for the Variational Autoencoder (VAE) family and is not only limited to pixel-wise similarity losses, but is common to all methods relying upon …

Deep Anomaly Detection With Deviation Networks, Guansong Pang, Chunhua Shen, Anton Van Den Hengel

Research Collection School Of Computing and Information Systems

Although deep learning has been applied to successfully address many data mining problems, relatively limited work has been done on deep learning for anomaly detection. Existing deep anomaly detection methods, which focus on learning new feature representations to enable downstream anomaly detection methods, perform indirect optimization of anomaly scores, leading to data-inefficient learning and suboptimal anomaly scoring. Also, they are typically designed as unsupervised learning due to the lack of large-scale labeled anomaly data. As a result, they are difficult to leverage prior knowledge (e.g., a few labeled anomalies) when such information is available as in many real-world anomaly detection …

Probabilistic Clustering Ensemble Evaluation For Intrusion Detection, Steven M. Mcelwee

CCE Theses and Dissertations

Intrusion detection is the practice of examining information from computers and networks to identify cyberattacks. It is an important topic in practice, since the frequency and consequences of cyberattacks continues to increase and affect organizations. It is important for research, since many problems exist for intrusion detection systems. Intrusion detection systems monitor large volumes of data and frequently generate false positives. This results in additional effort for security analysts to review and interpret alerts. After long hours spent reviewing alerts, security analysts become fatigued and make bad decisions. There is currently no approach to intrusion detection that reduces the workload …

Improving Service Level Of Free-Floating Bike Sharing Systems, Aritra Pal

USF Tampa Graduate Theses and Dissertations

Bike Sharing is a sustainable mode of urban mobility, not only for regular commuters but also for casual users and tourists. Free-floating bike sharing (FFBS) is an innovative bike sharing model, which saves on start-up cost, prevents bike theft, and offers significant opportunities for smart management by tracking bikes in real-time with built-in GPS. Efficient management of a FFBS requires: 1) analyzing its mobility patterns and spatio-temporal imbalance of supply and demand of bikes, 2) developing strategies to mitigate such imbalances, and 3) understanding the causes of a bike getting damaged and developing strategies to minimize them. All of these …

Deep Neural Networks With Confidence Sampling For Electrical Anomaly Detection, Norman L. Tasfi, Wilson A. Higashino, Katarina Grolinger, Miriam A. M. Capretz

Electrical and Computer Engineering Publications

The increase in electrical metering has created tremendous quantities of data and, as a result, possibilities for deep insights into energy usage, better energy management, and new ways of energy conservation. As buildings are responsible for a significant portion of overall energy consumption, conservation efforts targeting buildings can provide tremendous effect on energy savings. Building energy monitoring enables identification of anomalous or unexpected behaviors which, when corrected, can lead to energy savings. Although the available data is large, the limited availability of labels makes anomaly detection difficult. This research proposes a deep semi-supervised convolutional neural network with confidence sampling for …

Using Self-Organizing Maps For Computer Network Intrusion Detection, Manuel R. Parrachavez

Theses and Dissertations

Anomaly detection in user access patterns using artificial neural networks is a novel way of combating the ever-present concern of computer network intrusion detection for many entities around the world. Anomaly detection is a technique in network security in which a profile is built around a user's normal daily actions. The data collected for these profiles can be as following: file access attempts; failed login attempts; file creations; file access failures; and countless others. This data is collected and used as training data for a neural network. There are many types of neural networks, such as multi-layer feed-forward network; recurrent …

Model-Based Outlier Detection System With Statistical Preprocessing, D. Asir Antony Gnana Singh, E. Jebalamar Leavline

Journal of Modern Applied Statistical Methods

Reliability, lack of error, and security are important improvements to quality of service. Outlier detection is a process of detecting the erroneous parts or abnormal objects in defined populations, and can contribute to secured and error-free services. Outlier detection approaches can be categorized into four types: statistic-based, unsupervised, supervised, and semi-supervised. A model-based outlier detection system with statistical preprocessing is proposed, taking advantage of the statistical approach to preprocess training data and using unsupervised learning to construct the model. The robustness of the proposed system is evaluated using the performance evaluation metrics sum of squared error (SSE) and time to …

Lesinn: Detecting Anomalies By Identifying Least Similar Nearest Neighbours, Guansong Pang, Kai Ming Ting, David Albrecht

Research Collection School Of Computing and Information Systems

We introduce the concept of Least Similar Nearest Neighbours (LeSiNN) and use LeSiNN to detect anomalies directly. Although there is an existing method which is a special case of LeSiNN, this paper is the first to clearly articulate the underlying concept, as far as we know. LeSiNN is the first ensemble method which works well with models trained using samples of one instance. LeSiNN has linear time complexity with respect to data size and the number of dimensions, and it is one of the few anomaly detectors which can apply directly to both numeric and categorical data sets. Our extensive …

Exploring Discriminative Features For Anomaly Detection In Public Spaces, Shriguru Nayak, Archan Misra, Kasthuri Jeyarajah, Philips Kokoh Prasetyo, Ee-Peng Lim

Research Collection School Of Computing and Information Systems

Context data, collected either from mobile devices or from user-generated social media content, can help identify abnormal behavioural patterns in public spaces (e.g., shopping malls, college campuses or downtown city areas). Spatiotemporal analysis of such data streams provides a compelling new approach towards automatically creating real-time urban situational awareness, especially about events that are unanticipated or that evolve very rapidly. In this work, we use real-life datasets collected via SMU's LiveLabs testbed or via SMU's Palanteer software, to explore various discriminative features (both spatial and temporal - e.g., occupancy volumes, rate of change in topic{specific tweets or probabilistic distribution of …