Essential Virtual SAN (VSAN) (2^nd Ed.)
Administrator's Guide to VMware Virtual SAN

Plan, implement, and manage VMware’s radically simple, enterprise-class software-defined storage platform

VMware’s Virtual SAN has rapidly proven itself in environments ranging from hospitals to oil rigs to e-commerce platforms. Along the way, it has matured to offer unsurpassed features for data integrity, availability, and space efficiency. Virtual SAN 6.x makes all-flash storage practical for even more use cases, while radically simplifying IT operations and supporting the transition to hyper-converged infrastructures (HCI). Now, the authors of Essential Virtual SAN (VSAN) have thoroughly updated their definitive guide to this transformative technology. Writing for vSphere administrators, architects, and consultants, Cormac Hogan and Duncan Epping explain what Virtual SAN is, how it has evolved, what it now offers, and how to gain maximum value from it.

Hogan and Epping draw on unsurpassed experience shaping Virtual SAN and helping VMware customers deploy it. They offer expert insight into preparation, installation, configuration, policies, provisioning, clusters, and more. You’ll also find practical guidance for using its new Health and Performance Services to gain end-to-end visibility into infrastructure and resource consumption. Both an up-to-the-minute reference and hands-on tutorial, Essential Virtual SAN, Second Edition uses realistic examples to demonstrate the immense power of Virtual SAN 6.x. You’ll learn all you need to successfully plan and deploy the newest versions, and operate them smoothly and efficiently.

COVERAGE INCLUDES:

• Understanding the goals and concepts of Software-Defined Storage and Virtual SAN
• Meeting updated requirements for safe Virtual SAN 6.x implementation
• Architecting, installing, and configuring Virtual SAN for your unique environment
• Simplifying deployment with VM storage policies and provisioning
• Controlling availability, performance, and reliability
• Efficiently managing and maintaining Virtual SAN
• Providing resiliency and scale-out storage functionality
• Designing and sizing clusters (with examples)
• Using stretched clusters to address rigorous downtime requirements
• Applying valuable new features such as deduplication, compression, checksums, and encryption
• Using Health and Performance Services to troubleshoot hardware, configurations, and performance

Foreword by Christos Karamanolis xvii

About the Author xix

About the Technical Reviewers xxi

Acknowledgments xxiii

We Want to Hear from You! xxv

1 Introduction to VSAN 1

Software-Defined Datacenter 1

Software-Defined Storage 2

Hyper-Convergence/Server SAN Solutions 3

Introducing Virtual SAN 4

What Is Virtual SAN? 6

What Does VSAN Look Like to an Administrator? 9

Summary 12

2 VSAN Prerequisites and Requirements for Deployment 13

VMware vSphere 13

  ESXi 14

  Cache and Capacity Devices 14

  ESXi Boot Considerations 15

VSAN Requirements 15

  VMware Hardware Compatibility Guide 16

  VSAN Ready Nodes 16

  Storage Controllers 17

  Capacity Tier Devices 19

  Cache Tier Devices 21

Network Requirements 22

  Network Interface Cards 22

  Supported Virtual Switch Types 22

  Layer 2 or Layer 3 23

  VMkernel Network 23

  VSAN Network Traffic 24

  Jumbo Frames 24

  NIC Teaming 25

  Network I/O Control 25

VSAN Stretched Cluster 25

VSAN 2-Node Remote Office/Branch Office (ROBO) 26

  Firewall Ports 26

Summary 26

3 VSAN Installation and Configuration 29

VSAN Networking 29

VMkernel Network for VSAN 30

    VSAN Network Configuration: VMware Standard Switch 31

VSAN Network Configuration: vSphere Distributed Switch 32

  Step 1: Create the Distributed Switch 32

  Step 2: Create a Distributed Port Group 33

  Step 3: Build VMkernel Ports 34

Possible Network Configuration Issues 38

Network I/O Control Configuration Example 40

Design Considerations: Distributed Switch and Network I/O Control 42

  Scenario 1: Redundant 10 GbE Switch Without “Link Aggregation” Capability 43

  Scenario 2: Redundant 10 GbE Switch with Link Aggregation Capability 45

Creating a VSAN Cluster 48

vSphere HA 49

  vSphere HA Communication Network 49

  vSphere HA Heartbeat Datastores 50

  vSphere HA Admission Control 50

  vSphere HA Isolation Response 51

  vSphere HA Component Protection 51

The Role of Disk Groups 51

  Disk Group Maximums 52

  Why Configure Multiple Disk Groups in VSAN? 52

  Cache Device to Capacity Device Sizing Ratio 53

  Automatically Add Disks to VSAN Disk Groups 54

  Manually Adding Disks to a VSAN Disk Group 55

  Disk Group Creation Example 55

  VSAN Datastore Properties 58

Summary 59

4 VM Storage Policies on VSAN 61

Introducing Storage Policy-Based Management in a VSAN Environment 62

  Number of Failures to Tolerate 65

  Failure Tolerance Method 66

  Number of Disk Stripes Per Object 69

  IOPS Limit for Object 70

  Flash Read Cache Reservation 71

  Object Space Reservation 71

  Force Provisioning 71

  Disable Object Checksum 73

VASA Vendor Provider 73

  An Introduction to VASA 73

  Storage Providers 74

VSAN Storage Providers: Highly Available 75

  Changing VM Storage Policy On-the-Fly 75

  Objects, Components, and Witnesses 80

VM Storage Policies 80

  Enabling VM Storage Policies 81

  Creating VM Storage Policies 81

  Assigning a VM Storage Policy During VM Provisioning 81

Summary 82

5 Architectural Details 83

Distributed RAID 83

Objects and Components 86

  Component Limits 87

  Virtual Machine Storage Objects 88

  Namespace 89

  Virtual Machine Swap 90

  VMDKs and Deltas 90

  Witnesses and Replicas 90

  Object Layout 91

VSAN Software Components 94

  Component Management 95

  Data Paths for Objects 95

  Object Ownership 96

  Placement and Migration for Objects 96

  Cluster Monitoring, Membership, and Directory Services 97

  Host Roles (Master, Slave, Agent) 97

  Reliable Datagram Transport 98

On-Disk Formats 98

  Cache Devices 99

  Capacity Devices 99

VSAN I/O Flow 100

  Caching Algorithms 100

  The Role of the Cache Layer 100

  Anatomy of a VSAN Read on Hybrid VSAN 102

  Anatomy of a VSAN Read on All-Flash VSAN 103

  Anatomy of a VSAN Write on Hybrid VSAN 103

  Anatomy of a VSAN Write on All-Flash VSAN 104

  Retiring Writes to Capacity Tier on Hybrid VSAN 105

  Deduplication and Compression 105

  Data Locality 107

  Data Locality in VSAN Stretched Clusters 108

Storage Policy-Based Management 109

VSAN Capabilities 109

  Number of Failures to Tolerate Policy Setting 110

  Best Practice for Number of Failures to Tolerate 112

  Stripe Width Policy Setting 113

  RAID-0 Used When No Striping Specified in the Policy 117

  Stripe Width Maximum 119

  Stripe Width Configuration Error 120

  Stripe Width Chunk Size 121

  Stripe Width Best Practice 122

  Flash Read Cache Reservation Policy Setting 122

  Object Space Reservation Policy Setting 122

  VM Home Namespace Revisited 123

  VM Swap Revisited 123

  How to Examine the VM Swap Storage Object 124

  Delta Disk / Snapshot Caveat 126

  Verifying How Much Space Is Actually Consumed 126

  Force Provisioning Policy Setting 127

  Witnesses and Replicas: Failure Scenarios 127

  Data Integrity Through Checksum 130

  Recovery from Failure 131

  Problematic Device Handling 134

  What About Stretching VSAN? 134

Summary 135

6 VM Storage Policies and Virtual Machine Provisioning 137

Policy Setting: Number of Failures to Tolerate = 1 137

Policy Setting: Failures to Tolerate = 1, Stripe Width = 2 144

Policy Setting: Failures to Tolerate = 2, Stripe Width = 2 148

Policy Setting: Failures to Tolerate = 1, Object Space Reservation = 50% 152

Policy Setting: Failures to Tolerate = 1, Object Space Reservation = 100% 155

Policy Setting: RAID-5 157

Policy Setting: RAID-6 158

Policy Setting: RAID-5/6 and Stripe Width = 2 159

Default Policy 160

Summary 164

7 Management and Maintenance 165

Health Check 165

  Health Check Tests 165

  Proactive Health Checks 167

  Performance Service 168

Host Management 169

  Adding Hosts to the Cluster 169

  Removing Hosts from the Cluster 170

  ESXCLI VSAN Cluster Commands 171

Maintenance Mode 172

  Default Maintenance Mode/Decommission Mode 175

  Recommended Maintenance Mode Option for Updates and Patching 175

Disk Management 177

  Adding a Disk Group 177

  Removing a Disk Group 178

  Adding Disks to the Disk Group 179

  Removing Disks from the Disk Group 180

Wiping a Disk 182

  Blinking the LED on a Disk 183

  ESXCLI VSAN Disk Commands 184

Failure Scenarios 185

  Capacity Device Failure 185

  Cache Device Failure 186

  Host Failure 187

  Network Partition 188

  Disk Full Scenario 193

Thin Provisioning Considerations 194

vCenter Management 195

  vCenter Server Failure Scenario 196

  Running vCenter Server on VSAN 196

  Bootstrapping vCenter Server 197

Summary 199

8 Stretched Cluster 201

What is a Stretched Cluster? 201

Requirements and Constraints 203

Networking and Latency Requirements 205

New Concepts in VSAN Stretched Cluster 206

Configuration of a Stretched Cluster 208

Failure Scenarios 216

Summary 224

9 Designing a VSAN Cluster 225

Ready Node Profiles 225

Sizing Constraints 227

Cache to Capacity Ratio 228

Designing for Performance 229

  Impact of the Disk Controller 231

VSAN Performance Capabilities 235

Design and Sizing Tools 236

Scenario 1: Server Virtualization–Hybrid 237

  Determining Your Host Configuration 238

Scenario 2–Server Virtualization–All-flash 241

Summary 244

10 Troubleshooting, Monitoring, and Performance 245

Health Check 246

  Ask VMware 246

  Health Check Categories 247

  Proactive Health Checks 253

ESXCLI 256

  esxcli vsan datastore 256

  esxcli vsan network 257

  esxcli vsan storage 258

  esxcli vsan cluster 262

  esxcli vsan faultdomain 263

  esxcli vsan maintenancemode 264

  esxcli vsan policy 264

  esxcli vsan trace 267

  Additional Non-ESXCLI Commands for Troubleshooting VSAN 268

Ruby vSphere Console 275

  VSAN Commands 276

  SPBM Commands 300

Troubleshooting VSAN on the ESXi 303

  Log Files 304

  VSAN Traces 304

  VSAN VMkernel Modules and Drivers 305

Performance Monitoring 305

  Introducing the Performance Service 305

  ESXTOP Performance Counters for VSAN 308

  vSphere Web Client Performance Counters for VSAN 309

  VSAN Observer 310

Sample VSAN Observer Use Case 316

Summary 318

TOC, 9780134511665, 5/10/2016

Cormac Hogan is a Senior Staff Engineer in the Office of the CTO in the Storage and Availability business unit at VMware. Cormac was one of the first VMware employees at the EMEA headquarters in Cork, Ireland, back in 2005, and has previously held roles in VMware’s Technical Marketing, Integration Engineering and Support organizations. Cormac has written a number of storage-related white papers and has given numerous presentations on storage best practices and new features. Cormac is the owner of CormacHogan.com, a blog site dedicated to storage and virtualization.

He can be followed on twitter @CormacJHogan.

Duncan Epping is a Chief Technologist working for VMware in the Office of CTO of the Storage and Availability business unit. Duncan is responsible for ensuring VMware’s future innovations align with essential customer needs, translating customer problems to opportunities, and function as the global lead evangelist for Storage and Availability. Duncan specializes in Software Defined Storage, hyper-converged infrastructures and business continuity/disaster recovery solutions. He has four patents pending and one granted on the topic of availability, storage and resource management. Duncan is the author/owner of VMware Virtualization blog Yellow-Bricks.com and has various books on the topic of VMware including the “vSphere Clustering Deepdive” series.

He can be followed on twitter @DuncanYB.