Distributed Storage Networks

The worldwide market for SAN and NAS storage is anticipated to grow from US $2 billion in 1999 to over $25 billion by 2004. As business-to-business and business-to-consumer e-commerce matures, even greater demands for management of stored data will arise. With the rapid increase in data storage requirements in the last decade, efficient management of stored data becomes a necessity for the enterprise. A recent UC-Berkeley study predicts that 150,000 terabytes of disk storage will be shipped in 2003. Most financial, insurance, healthcare, and telecommunications institutions are in the process of implementing storage networks that are distributed to some degree. For these institutions, data integrity is critical, and they will spend much time and money on planning. One of the primary obstacles to implementing a storage network cited by enterprise IT managers is a lack of knowledge about storage networking technology and the specific issues involved in extending a Storage Area Network (SAN) or Network Attached Storage (NAS) over the Metropolitan Area Networks (MAN) or Wireless Area Networks (WAN). Distributed Storage Networks : Architecture, Protocols and Management addresses the "terminology gap" between enterprise network planners and telecommunications engineers, who must understand the transport requirements of storage networks in order to implement distributed storage networks. Jepsen comprehensively provides IT managers, planners, and telecommunications professionals with the information they need in order to choose the technologies best suited for their particular environment. * Addresses a hot topic that will become increasingly important in the coming years * Enables high-level managers and planners to make intelligent decisions about network needs. * Includes example network configurations providing solutions to typical user scenarios * Fills the "terminology gap" between enterprise network managers and telecommunications engineers who must understand the transport requirements of storage networks in order to implement distributed storage area networks A fundamental resource for all network managers, planners and network design engineers, as well as telecommunications engineers and engineering, computer science, and information technology students.

Cloud storage services and NoSQL systems typically offer only "Eventual Consistency", a rather weak guarantee covering a broad range of potential data consistency behavior. The degree of actual (in-)consistency, however, is unknown. This work presents novel solutions for determining the degree of (in-)consistency via simulation and benchmarking, as well as the necessary means to resolve inconsistencies leveraging this information.

Unlike networking technology, where there is already a great deal of literature available, many professionals still need to understand the basic building blocks of storage networking. This book provides vendor-neutral, independent analysis and terminology.

Organizations today depend heavily on their data. Even short periods of data outages can be expensive and result in loss of productivity, as well as financial consequences, while permanent data loss can be catastrophic. Therefore, reliability and means to efficiently store and access such data is an important component of most large organizations' IT infrastructure. Much of this data is still stored in the most versatile format, the 'flat file'. This eBook provides both an academic and historic perspective on the development of distributed file systems and details some of the core algorithms, such as quorum protocols that are used in distributed storage systems. This book can be used as a short, stand-alone introduction to the field or as a resource for an academic course in the topic.

Market_Desc: The book provides basic application information key for systems administrators, database administrators and managers who need to know about the networking aspects of their systems. As well as systems architects, network managers, information management directors and decision makers.This book also supports applications for graduate students and other relevant courses in the field. Special Features: · Hot topic that will become increasingly important in the coming years· First book to focus on using rather than building storage networks, and how to solve problems· Looking beyond technology and showing the With CD benefits of storage networks· Covers fibre channel SAN, Network Attached Storage, iSCSI and InfiniBand technologies· Contains several case studies (e.g. the example of a travel portal, protecting a critical database)· Endorsed by the Storage Networking Industry Association· Written by very experienced professionals who tailored the book specifically to meet customer needs About The Book: The authors have hands-on experience of network storage hardware and software, they teach customers about concrete network storage products, they understand the concepts behind storage networks, and show customers how storage networks address their business needs. They know which questions their readers will ask and what they need to know to do their day-to-day job as efficiently as possible, both those with no SAN experience and those with SAN experience.

Cloud storage services and NoSQL systems typically offer only ""Eventual Consistency"", a rather weak guarantee covering a broad range of potential data consistency behavior. The degree of actual (in-)consistency, however, is unknown. This work presents novel solutions for determining the degree of (in-)consistency via simulation and benchmarking, as well as the necessary means to resolve inconsistencies leveraging this information. This work was published by Saint Philip Street Press pursuant to a Creative Commons license permitting commercial use. All rights not granted by the work's license are retained by the author or authors.

* The emphasis of this book will be on detailed practicality. Most of the SAN books provide a theoretical treatment of the technology from a top-down perspective. This book will be written from the perspective of "from the ground up". * Relates specific technology offerings to particular application areas. Email stores, Image stores, Video Production and RDBMS disk are used as specific case studies to show how the hardware, firmware, and interconnects are set up and used. * SAN technology is ready to move out of the glass house and large scale storage is becoming applicable to even dedicated purposes. This represents an increase in the potential audience for a book on SANs and, of course, remains highly useful for the administrators and centralized technical staff responsible for backups, recoverability, and availability.

We are dependent upon data in many aspects of our lives. Much of this data is stored and managed by distributed storage systems that run in data centers, powering many modern applications such as e-commerce, photo sharing, video streaming, search, social networking, messaging, collaborative editing, and even health-care and financial services. A distributed storage system stores copies of a piece of data on many nodes for fault-tolerance: even when a few nodes fail, the system can still provide access to data. Each of these nodes depends upon a local storage stack to safely store and manage user data. The local storage stack is complex, consisting of many hardware and software components. Due to this complexity, the storage layer is a place for many potential problems to arise. This dissertation examines the reliability and performance challenges that arise the interaction points between a distributed system and the local storage stack. In the first part of this thesis, we study how distributed storage systems react to storage faults: cases where the storage device may return corrupted data or errors. We focus on replicated state machine systems, an important class of distributed systems. We find that none of the existing approaches used in current systems can safely handle storage faults, leading to data loss and unavailability. Using the insights gained in our study, we design corruption-tolerant replication (CTRL), a protocol-aware recovery approach for RSM systems. CTRL exploits protocol-specific knowledge of how RSM systems operate, to ensure safety and high availability in the presence of storage faults without impacting performance. In the second part, we study the performance and reliability properties of replication protocols used by distributed systems. We find there exists a dichotomy with respect to how and where current approaches store system state. One approach writes data to the storage stack synchronously, whereas the other buffers the data in volatile memory. The choice of whether data is written synchronously to the storage device or not greatly influences the system's robustness to crash failures and its performance. We show that existing approaches either provide robustness to crashes or performance, but not both. Thus, we introduce situation-aware updates and crash recovery, a dynamic protocol that, depending upon the situation, writes either synchronously or asynchronously to the storage devices, achieving both strong reliability and high performance. In the final part of this thesis, we study the effects of file-system crash behaviors in distributed storage systems. We build protocol-aware crash explorer or PACE, a tool that can model and reason about file-system crash behaviors in distributed systems under a special correlated crash failure scenario. Our study reveals that the correctness of update and recovery protocols of many distributed systems hinges upon how the local file-system state is updated by each replica. We perform a detailed analysis of the vulnerabilities, showing their serious consequences and prevalence on commonly used file systems. We finally point to possible solutions to the problems discovered.

This book was originally published in 1995. At the time of publication, distributed file systems were monolithic and only supported single file abstractions. Network storage devices needed to be able to accommodate emerging information media such as digital audio and video, with data radically different in characteristics to traditional text and binary that file systems were optimised for. By combining emerging and traditional media, information could be recorded and presented in the most suitable way, and the value of a piece of information could be further enhanced by linking together related pieces. However composite data and cross-reference between data items raised a number of system issues that had not been addressed properly before. In this book Dr Lo defined a multi-service storage architecture that could meet the needs of existing and emerging applications and support multiple file abstractions. He also explored a number of related design issues.

Providing a shared memory abstraction in distributed systems is a powerful tool that can simplify the design and implementation of software systems for networked platforms. This enables the system designers to work with abstract readable and writable objects without the need to deal with the complexity and dynamism of the underlying platform. The key property of shared memory implementations is the consistency guarantee that it provides under concurrent access to the shared objects. The most intuitive memory consistency model is atomicity because of its equivalence with a memory system where accesses occur serially, one at a time. Emulations of shared atomic memory in distributed systems is an active area of research and development. The problem proves to be challenging, and especially so in distributed message passing settings with unreliable components, as is often the case in networked systems. We present several approaches to implementing shared memory services with the help of replication on top of message-passing distributed platforms subject to a variety of perturbations in the computing medium.