Flash-Based Storage Management In Cloud Computing Datacenter Infrastructures, Zhengyu Yang
Dec 2017
Flash-Based Storage Management In Cloud Computing Datacenter Infrastructures, Zhengyu Yang
Zhengyu Yang
A basic credendum of cloud computing can be summarized as: user devices are light terminals to assign jobs and gather results, while those heavy computations are conducted on remote distributed server clusters. This light-terminal-heavy-server structure makes high availability no longer an option, but a requirement in today’s datacenters. Furthermore, when bringing compute and storage capabilities into balance, we find that the biggest challenge here is closing the gap between compute and storage performance to shift storage’s curve back towards Moore’s law. In detail, the time consumed to wait for I/Os is the main cause of idling and wasting CPU resources, …
New Yarn Non-Exclusive Resource Management Scheme Through Opportunistic Idle Resource Assignment, Zhengyu Yang, Yi Yao, Han Gao, Jiayin Wang, Ningfang Mi, Bo Sheng
Dec 2017
New Yarn Non-Exclusive Resource Management Scheme Through Opportunistic Idle Resource Assignment, Zhengyu Yang, Yi Yao, Han Gao, Jiayin Wang, Ningfang Mi, Bo Sheng
Zhengyu Yang
Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and …
Autopath: Harnessing Parallel Execution Paths For Efficient Resource Allocation In Multi-Stage Big Data Frameworks, Han Gao, Zhengyu Yang, Janki Bhimani, Teng Wang, Jiayin Wang, Ningfang Mi, Bo Sheng
Dec 2016
Autopath: Harnessing Parallel Execution Paths For Efficient Resource Allocation In Multi-Stage Big Data Frameworks, Han Gao, Zhengyu Yang, Janki Bhimani, Teng Wang, Jiayin Wang, Ningfang Mi, Bo Sheng
Zhengyu Yang
Due to the flexibility of data operations and scalability of in-memory cache, Spark has revealed the potential to become the standard distributed framework to replace Hadoop for data-intensive processing in both industry and academia. However, we observe that the built-in scheduling algorithms in Spark (i.e., FIFO and FAIR) are not optimized for the applications with multiple parallel and independent branches in stages. Specifically, the child stage needs to wait and collect data from all its parent branches, but this wait has no guaranteed upper bound since it is tightly coupled with each branch’s workload characteristic, stage order, and their corresponding …
Finite Element Simulation Of Prevention Thermal Cracking In Mass Concrete, Juncai Xu, Zhengzhong Shen, Song Yang, Xin Xie, Zhengyu Yang
Dec 2016
Finite Element Simulation Of Prevention Thermal Cracking In Mass Concrete, Juncai Xu, Zhengzhong Shen, Song Yang, Xin Xie, Zhengyu Yang
Zhengyu Yang
Mass concrete structures play a very important role in civil engineering. The cracking of concrete is regarded as one of the biggest engineering problems. Therefore, it is very necessary for the cracking of mass concrete to do the control analysis. Some factors should be considered in mass concrete crack control analysis, mainly including the heat releases model of concrete, the mechanical model to the concrete, the process of temperature control in the pipe model. Differential evolution algorithm and equivalent algorithm are adopted to solve the coefficient of adiabatic temperature and cool water effect. In the paper, stress field calculation, back …
Fim: Performance Prediction For Parallel Computation In Iterative Data Processing Applications, Janki Bhimani, Ningfang Mi, Miriam Leeser, Zhengyu Yang
Dec 2016
Fim: Performance Prediction For Parallel Computation In Iterative Data Processing Applications, Janki Bhimani, Ningfang Mi, Miriam Leeser, Zhengyu Yang
Zhengyu Yang
Predicting performance of an application running on high performance computing (HPC) platforms in a cloud environment is increasingly becoming important because of its influence on development time and resource management. However, predicting the performance with respect to parallel processes is complex for iterative, multi-stage applications. This research proposes a performance approximation approach FiM to model the computing performance of iterative, multi-stage applications running on a master-compute framework. FiM consists of two key components that are coupled with each other: 1) Stochastic Markov Model to capture non-deterministic runtime that often depends on parallel resources, e.g., number of processes. 2) Machine Learning …
Ea2s2: An Efficient Application-Aware Storage System For Big Data Processing In Heterogeneous Clusters, Teng Wang, Jiayin Wang, Son Nam Nguyen, Zhengyu Yang, Ningfang Mi, Bo Sheng
Dec 2016
Ea2s2: An Efficient Application-Aware Storage System For Big Data Processing In Heterogeneous Clusters, Teng Wang, Jiayin Wang, Son Nam Nguyen, Zhengyu Yang, Ningfang Mi, Bo Sheng
Zhengyu Yang
Big data processing frameworks such as Hadoop have been widely adopted to process a large volume of data. A lot of prior work has focused on the allocation of resources and the execution order of jobs/tasks to improve the performance in a homogeneous cluster. In this paper, we investigate storage layer design in a heterogeneous system considering a new type of bundled jobs where the input data and associated application jobs are submitted in a bundle. Our goal is to break the barrier between resource management and the underlying storage layer, and improve data locality, an important performance factor for …
Accelerating Big Data Applications Using Lightweight Virtualization Framework On Enterprise Cloud, Janki Bhimani, Zhengyu Yang, Miriam Leeser, Ningfang Mi
Dec 2016
Accelerating Big Data Applications Using Lightweight Virtualization Framework On Enterprise Cloud, Janki Bhimani, Zhengyu Yang, Miriam Leeser, Ningfang Mi
Zhengyu Yang
Hypervisor-based virtualization technology has been successfully used to deploy high-performance and scalable infrastructure for Hadoop, and now Spark applications. Container-based virtualization techniques are becoming an important option, which is increasingly used due to their lightweight operation and better scaling when compared to Virtual Machines (VM). With containerization techniques such as Docker becoming mature and promising better performance, we can use Docker to speed-up big data applications. However, as applications have different behaviors and resource requirements, before replacing traditional hypervisor-based virtual machines with Docker, it is important to analyze and compare performance of applications running in the cloud with VMs and …
Enhancing Ssds With Multi-Stream: What? Why? How?, Janki Bhimani, Jingpei Yang, Zhengyu Yang, Ningfang Mi, N. H. V. Krishna Giri, Rajinikanth Pandurangan, Changho Choi, Vijay Balakrishnan
Dec 2016
Enhancing Ssds With Multi-Stream: What? Why? How?, Janki Bhimani, Jingpei Yang, Zhengyu Yang, Ningfang Mi, N. H. V. Krishna Giri, Rajinikanth Pandurangan, Changho Choi, Vijay Balakrishnan
Zhengyu Yang
The adoption of SSDs has become very prominent, but they still suffer from challenges to control write amplification. Traditional SSDs have single active append point where new data writes can be stored. Data of different lifetime stored together causes high write amplification. Recently, multi-stream SSDs are developed that allows multiple active append points. These multiple active append points can be used to store data of different lifetime in different locations within SSD. Such a data placement according to the lifetime of data would considerably reduce internal write amplification of SSD. For using multistream SSDs it is required to attach stream-id …
Seina: A Stealthy And Effective Internal Attack In Hadoop Systems, Jiayin Wang, Teng Wang, Zhengyu Yang, Ying Mao, Ningfang Mi, Bo Sheng
Dec 2016
Seina: A Stealthy And Effective Internal Attack In Hadoop Systems, Jiayin Wang, Teng Wang, Zhengyu Yang, Ying Mao, Ningfang Mi, Bo Sheng
Zhengyu Yang
Big data processing frameworks such as Hadoop [1] have been widely adopted in the past few years. However, the security issues in such large scale systems have not been well studied yet. While most of the prior work is focused on the data privacy and protection, this paper investigates a potential attack from a compromised internal node against the overall system performance. We explore the vulnerabilities of the existing Hadoop system, and develop an effective attack launched from the compromised node that can significantly degrade the data processing performance of the cluster without being detected and blacklisted for job execution. …
An Algorithm For Non-Steady Thermal Dynamics Finite Element Simulation And Differential Evolution, Juncai Xu, Zhenzhong Shen, Qingwen Ren, Xin Xie, Zhengyu Yang
Dec 2016
An Algorithm For Non-Steady Thermal Dynamics Finite Element Simulation And Differential Evolution, Juncai Xu, Zhenzhong Shen, Qingwen Ren, Xin Xie, Zhengyu Yang
Zhengyu Yang
Thermodynamic parameters of concrete are the significant condition in the preventing mass concrete cracking computation. The concrete inversion problem of thermodynamic parameters is a multi-parameter optimization problem. Differential Evolution is one of the optimization method of evolutionary and developed from GA (genetic algorithm). In this paper, non-steady temperature field finite element simulation and DE (Differential Evolution) combined together to establish DE inverse solution for concrete thermodynamic parameter including equivalent heat source method to realize water-pipe cooling simulation. The procedure was implemented in mode with high computational efficiency and being high accuracy. It is an effective way to select thermal parameters …
Automatic And Scalable Data Replication Manager In Distributed Computation And Storage Infrastructure Of Cyber-Physical Systems, Zhengyu Yang, Janki Bhimani, Jiayin Wang, David Evans, Ningfang Mi
Dec 2016
Automatic And Scalable Data Replication Manager In Distributed Computation And Storage Infrastructure Of Cyber-Physical Systems, Zhengyu Yang, Janki Bhimani, Jiayin Wang, David Evans, Ningfang Mi
Zhengyu Yang
Cyber-Physical System (CPS) is a rising technology that utilizes computation and storage resources for sensing, processing, analysis, predicting, understanding of field-data, and then uses communication resources for interaction, intervene, and interface management, and finally provides control for systems so that they can inter-operate, evolve, and run in a stable evidence-based environment. There are two major demands when building the storage infrastructure for a CPS cluster to support above-mentioned functionalities: (1) high I/O and network throughput requirements during runtime, and (2) low latency demand for disaster recovery. To address challenges brought by these demands, in this paper, we propose a complete …
Efficient Data Caching Management In Scalable Multi-Stage Data Processing Systems, Jiayin Wang, Zhengyu Yang, David Evans
Dec 2016
Efficient Data Caching Management In Scalable Multi-Stage Data Processing Systems, Jiayin Wang, Zhengyu Yang, David Evans
Zhengyu Yang
According to some example embodiments, a method includes: receiving, by a processor, from a data source, a processing profile comprising input data blocks and a plurality of operations for executing using the input data blocks; executing, by the processor, one or more of the operations of the processing profile to generate a new output data after each of the executed one or more operations; storing, by the processor, the new output data from at least one of the one or more operations as intermediate cache data; and transmitting, by the processor, the new output data from a final operation from …
Automatic Data Replica Manager In Distributed Caching And Data Processing System, Zhengyu Yang, Jiayin Wang, David Evans
Dec 2016
Automatic Data Replica Manager In Distributed Caching And Data Processing System, Zhengyu Yang, Jiayin Wang, David Evans
Zhengyu Yang
A method of data storage includes determining a latency distance from a primary node to each of two or more replica nodes, choosing a preferred replica node of the two or more replica nodes based on the determined latency distances, and write-caching data into the preferred replica node.
Adaptive Caching Replacement Manager With Dynamic Updating Granularities And Partitions For Shared Flash-Based Storage System, Zhengyu Yang, David Evans, Jiayin Wang
Dec 2016
Adaptive Caching Replacement Manager With Dynamic Updating Granularities And Partitions For Shared Flash-Based Storage System, Zhengyu Yang, David Evans, Jiayin Wang
Zhengyu Yang
A method of adjusting temporal and spatial granularities associated with operation of a virtualized file system, the method including analyzing past workloads of a plurality of virtual machines associated with the virtualized file system, and adjusting the temporal and spatial granularities to be similar to average re-access temporal and spatial distances of data sets corresponding to the past workloads.
Grem: Dynamic Ssd Resource Allocation In Virtualized Storage Systems With Heterogeneous Io Workloads, Zhengyu Yang, Jianzhe Tai, Janki Bhimani, Jiayin Wang, Ningfang Mi, Bo Sheng
Dec 2015
Grem: Dynamic Ssd Resource Allocation In Virtualized Storage Systems With Heterogeneous Io Workloads, Zhengyu Yang, Jianzhe Tai, Janki Bhimani, Jiayin Wang, Ningfang Mi, Bo Sheng
Zhengyu Yang
In a shared virtualized storage system that runs VMs with heterogeneous IO demands, it becomes a problem for the hypervisor to cost-effectively partition and allocate SSD resources among multiple VMs. There are two straightforward approaches to solving this problem: equally assigning SSDs to each VM or managing SSD resources in a fair competition mode. Unfortunately, neither of these approaches can fully utilize the benefits of SSD resources, particularly when the workloads frequently change and bursty IOs occur from time to time. In this paper, we design a Global SSD Resource Management solution - GREM, which aims to fully utilize SSD …
A Fresh Perspective On Total Cost Of Ownership Models For Flash Storage In Datacenters, Zhengyu Yang, Manu Awasthi, Mrinmoy Ghosh, Ningfang Mi
Dec 2015
A Fresh Perspective On Total Cost Of Ownership Models For Flash Storage In Datacenters, Zhengyu Yang, Manu Awasthi, Mrinmoy Ghosh, Ningfang Mi
Zhengyu Yang
Recently, adoption of Flash based devices has become increasingly common in all forms of computing devices. Flash devices have started to become more economically viable for large storage installations like datacenters, where metrics like Total Cost of Ownership (TCO) are of paramount importance. Flash devices suffer from write amplification (WA), which, if unaccounted, can substantially increase the TCO of a storage system. In this paper, we develop a TCO model for Flash storage devices, and then plug a Write Amplification (WA) model of NVMe SSDs we build based on empirical data into this TCO model. Our new WA model accounts …
Understanding Performance Of I/O Intensive Containerized Applications For Nvme Ssds, Janki Bhimani, Jingpei Yang, Zhengyu Yang, Ningfang Mi, Qiumin Xu, Manu Awasthi, Rajinikanth Pandurangan, Vijay Balakrishnan
Dec 2015
Understanding Performance Of I/O Intensive Containerized Applications For Nvme Ssds, Janki Bhimani, Jingpei Yang, Zhengyu Yang, Ningfang Mi, Qiumin Xu, Manu Awasthi, Rajinikanth Pandurangan, Vijay Balakrishnan
Zhengyu Yang
Our cloud-based IT world is founded on hypervisors and containers. Containers are becoming an important cornerstone, which is increasingly used day-by-day. Among different available frameworks, docker has become one of the major adoptees to use containerized platform in data centers and enterprise servers, due to its ease of deploying and scaling. Further more, the performance benefits of a lightweight container platform can be leveraged even more with a fast back-end storage like high performance SSDs. However, increase in number of simultaneously operating docker containers may not guarantee an aggregated performance improvement due to saturation. Thus, understanding performance bottleneck in a …
Autoreplica: Automatic Data Replica Manager In Distributed Caching And Data Processing Systems, Zhengyu Yang, Jiayin Wang, David Evans, Ningfang Mi
Dec 2015
Autoreplica: Automatic Data Replica Manager In Distributed Caching And Data Processing Systems, Zhengyu Yang, Jiayin Wang, David Evans, Ningfang Mi
Zhengyu Yang
Nowadays, replication technique is widely used in data center storage systems for large scale Cyber-physical Systems (CPS) to prevent data loss. However, side-effect of replication is mainly the overhead of extra network and I/O traffics, which inevitably downgrades the overall I/O performance of the cluster. To effectively balance the trade-off between I/O performance and fault tolerance, in this paper, we propose a complete solution called “AutoReplica” – a replica manager in distributed caching and data processing systems with SSD-HDD tier storages. In detail, AutoReplica utilizes the remote SSDs (connected by high speed fibers) to replicate local SSD caches to protect …
Online Flash Resource Migration, Allocation, Retire And Replacement Manager Based On A Cost Of Ownership Model, Zhengyu Yang, Mrinmoy Ghosh, Manu Awasthi, Vijay Balakrishnan
Dec 2015
Online Flash Resource Migration, Allocation, Retire And Replacement Manager Based On A Cost Of Ownership Model, Zhengyu Yang, Mrinmoy Ghosh, Manu Awasthi, Vijay Balakrishnan
Zhengyu Yang
A method for migrating disks includes: dividing a disk pool including a plurality of disks into a random zone and a sequential zone based on sequentiality and randomness of workloads running on the plurality of disks; monitoring a status of each disk in the disk pool based on a total cost of ownership (TCO); migrating one or more workloads of an overheated disk to an idle disk based on the status of each disk. The overheated disk has a first TCO higher than a migration threshold, and the idle disk has a second TCO lower than an idling threshold.
Online Flash Resource Allocation Manager Based On Tco Model, Zhengyu Yang, Mrinmoy Ghosh, Manu Awasthi, Vijay Balakrishnan
Dec 2015
Online Flash Resource Allocation Manager Based On Tco Model, Zhengyu Yang, Mrinmoy Ghosh, Manu Awasthi, Vijay Balakrishnan
Zhengyu Yang
A method for allocating workloads based on a total cost of ownership (TCO) model includes receiving a workload; estimating a cost for allocating the workload to each disk of disks in a disk pool based on a TCO model; determining a disk among the disks in the disk pool that minimizes a TCO; and allocating the workload to the disk. The TCO model incorporates a plurality of cost factors for estimating costs for each disk in the disk pool for allocating the workload.
Memory Device Having A Translation Layer With Multiple Associative Sectors, Zhengyu Yang, Sina Hassani, Manu Awasthi
Dec 2015
Memory Device Having A Translation Layer With Multiple Associative Sectors, Zhengyu Yang, Sina Hassani, Manu Awasthi
Zhengyu Yang
A method for providing a memory translation layer includes: receiving write request streams from a host computer; selectively storing each write request stream into a sequential zone, a K-associative zone, and a random zone of log blocks of a nonvolatile memory based on the characteristics. A first group of the write request streams that are sequential and start from a header page of a log block are stored in the sequential zone. A second group of the write request streams that are sequential but do not start from a header page of a log block are stored in the K-associative …
Improving Flash Resource Utilization At Minimal Management Cost In Virtualized Flash-Based Storage Systems, Zhengyu Yang, Jianzhe Tai, Deng Liu, Xiaoyun Zhu, Jack Lo, Ningfang Mi
Dec 2014
Improving Flash Resource Utilization At Minimal Management Cost In Virtualized Flash-Based Storage Systems, Zhengyu Yang, Jianzhe Tai, Deng Liu, Xiaoyun Zhu, Jack Lo, Ningfang Mi
Zhengyu Yang
Effectively leveraging Flash resources has emerged as a highly important problem in enterprise storage systems. One of the popular techniques today is to use Flash as a secondary-level host-side cache in the virtual machine environment. Although this approach delivers IO acceleration for VMs’ IO workloads, it might not be able to fully exploit the outstanding performance of Flash and justify the high cost-per-GB of Flash resources. In this paper, we design new VMware Flash Resource Managers (VFRM and GLB-VFRM) under the consideration of both performance and the incurred cost for managing Flash resources. Specifically, VFRM and GLB-VFRM aim to maximize …
Improving Virtual Machine Migration Via Deduplication, Jake Roemer, Mark Groman, Zhengyu Yang, Yufeng Wang, Chiu C. Tan, Ningfang Mi
Dec 2013
Improving Virtual Machine Migration Via Deduplication, Jake Roemer, Mark Groman, Zhengyu Yang, Yufeng Wang, Chiu C. Tan, Ningfang Mi
Zhengyu Yang
For this study the techniques of virtual machine migration are understood and the affects deduplication has on migration are evaluated. The benefits of using deduplication and compression on virtual machines show in the metric of space saved during migrating. Deduplication is computationally expensive so we evaluate how to group virtual machines with similar elements in order to improve migration. From this study, grouping virtual machines based on similar elements improves the overhead from deduplication and compression but estimates which virtual machines are best grouped together.
