Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 STO1479BU vsan Beyond the Basics Sumit Lahiri Product Line Manager Eric Knauft Staff Engineer #VMworld #STO1479BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. 2

3 Agenda 1 The world of Objects 2 Life of vsan Component 3 The 4 Rs of vsan 4 Multi-Level Fault Domains 5 All Flash I/O Flow 3

4 The World of Objects

5 Disk layout in host Cache Capacity vsan Datastore Disk groups contribute to single vsan datastore in vsphere cluster disk group disk group disk group disk group disk group Max 64 nodes Min 2 nodes (ROBO) Max 5 Disk Groups per host 2 Tiers per Disk Group

6 Creating vm, creates several objects in the background (VMDK) Virtual Disk VM home namespace: VMX, log files Virtual memory swap objects 6

7 From VM to components (Object) (components) (blocks) VMworld 2017 Component Component Component Component Content: Not for publication (Max Size: 255 GB) (in low MBs) 7

8 Fault Domains vsphere vsan Host Racks Sites 8

9 Failures to Tolerate (FTT) Always in context to fault domains vsphere vsan Host Racks Sites Failures to Tolerate Failures to Tolerate Failures to Tolerate 9

10 Failures to Tolerate (FTT) FTT implies host failures to tolerate if fault domain is not mentioned vsphere vsan vsphere vsan vsphere vsan FTT=1 FTT=2 FTT=3 10

11 Failures to Tolerate (FTT) can be Nested Survive one site failure and one host failure on the other site vsphere vsan Host Racks Sites 11

12 Fault Tolerance Methods

13 Failures Tolerate Method (FTM) FTT=1 FTT=2 FTT=3 vsphere vsan vsphere vsan vsphere vsan RAID-1. 2bytes/byte. 3bytes/byte 4bytes/byte. 1.3 bytes/byte RAID-5. X X 1.5 bytes/byte RAID-6 X. X 13

14 FTT = Failures to Tolerate FTM = Fault Tolerance Method

15 Notation

16 Object is associated with underlying policy (VMDK) Policy: 1. Failures to Tolerate 2. Fault Tolerance Method VMworld 2017 Content: Not for publication 16

17 Policy dictates how objects are managed (VMDK) Policy: 1. Failures to Tolerate (FTT) 2. Fault Tolerance Method (FTM) VMworld 2017 FTT =1, FTM = RAID-1, Stripe Width >2 Replica (stripes) C1 C2. (components) (VMDK) Replica Content: Not for publication (stripes) C1 C2. (components) 17

18 RAID Abstraction Model Replica C1 C2. (components) (stripes) (VMDK) Replica (stripes) FTT =1, FTM = RAID-1, Stripe Width >2 No witness (RAID-1) R1 (VMDK) (RAID-0) (RAID-0) R0 R0... C1 C2 C1 C2 C1 C2 (components) (components) (components) 18

19 FTT=1,FTM=RAID-1, comparison with stripe and without stripes (no striping) C 250GB (component) 250GB (RAID-1) R1 (VMDK) C (no striping) (component) No witness No witness 250GB 1TB (RAID-1) R1 (VMDK) (RAID-0) (RAID-0) R0 R0.. C1 C2 C1 C2 (components) (components) 250 GB 1 TB 19

20 vsan managed as bunch of components C C vsan Datastore C C C C components

21 Each replica on different Fault Domain (e.g. host) R0 (RAID-0) (RAID-1) R1 (VMDK) R0 (RAID-0) FTT =2, FTM = RAID-1, Stripe Width = 2 R0 (RAID-0) C1 C2 C1 C2 C1 C2 (components) (components) (components) 21

22 Each component is commonly placed on a different host R0 (RAID-0) (RAID-1) R1 (VMDK) R0 (RAID-0) FTT =2, FTM = RAID-1, Stripe Width = 2 R0 (RAID-0) C1 C2 C1 C2 C1 C2 (components) (components) (components) 22

23 Can we survive 2 host failures with 3 hosts? R0 (RAID-0) (RAID-1) R1 (VMDK) R0 (RAID-0) FTT =2, FTM = RAID-1, Stripe Width = 2 R0 (RAID-0) C1 C2 C1 C2 C1 C2 (components) (components) (components) 23

24 Liveness = Availability && Quorum

25 Quorum: In the event of cluster partition, which partition shall proceed? N hosts... partition-01 M hosts... partition-02 25

26 Quorum: The partition with the higher Votes proceed N votes N hosts... partition-01 M votes M hosts... partition-02 Cluster members participate in voting 26

27 If M > N, Partition-2 proceeds N votes N hosts... partition-01 partition-02 proceeds M votes M hosts... partition-02 Cluster members participate in voting 27

28 Voting FTT=1 and FTM = RAID-1

29 Quorum is calculated on a per object basis No witness C (component) (RAID-1) R1 (VMDK) 1 1 C (component) Each component participates in voting With two components, this sums to even number of votes 29

30 Add witness for Tier breaker vote (votes) 1 C (component) (RAID-1) R1 (votes) (VMDK) 1 C (component) 1 (votes) (witness) W Witness is added as Tier breaker vote Acts as an observer which component has latest data 30

31 For VMDK-A, partition-2 has higher votes (votes) 1 N hosts... (votes) 1 (RAID-1) R1 C (component) (VMDK-A) 1 M hosts... partition-01 partition-02 proceeds C (component) W 1 (witness) (votes) C C W (votes) 1 1 (votes) 31

32 General Case: Different objects proceed on different partition (votes) (votes) 1 1 (RAID-1) R1 C (component) (component) N hosts... (VMDK-B) (VMDK-A) 1 C W 1 (votes) (RAID-1) (witness) (votes) M hosts... C C W C W C (votes) 1 (votes) 1 1 C (component) R1 1 C (component) W 1 (witness) (votes) partition-01 proceeds for VMDK-B partition-02 proceeds for VMDK-A 32

33 Components can be classified as data component and witness component (no striping) (1 vote) D (data component) (RAID-1) R1 (1 vote) (VMDK) 1 1 D (no striping) (data component) 1 (1 vote) W (witness component)

34 Min count of hosts required for survive N host failures?

35 Minimum 2N+1 hosts required to survive N host failures N hosts = N shares of votes... partition-01 If each host represents same share of vote Wining partition would require a minimum of N+1 hosts (N +1) hosts = (N+1) shares of vote partition-02 is winning partition Minimum size of cluster = 2N+1 hosts to survive N host failures 35

36 Min cluster size is determined by meeting Liveness requirement Liveness = (Quorum) && (Availability) Min of hosts in cluster = Max (Min hosts for Quorum, Min hosts for Availability) 36

37 Examples FTT =1, FTM = RAID-1 Min host for availability = 2 Min host of Quorum = 2N+1 = 3 Min cluster size =3 FTT=2, FTM = RAID-1 Min host for availability = 3 Min host for Quorum = 2N+1 =5 Min cluster size =5 37

38 Examples of Liveness (Quorum + Availability)

39 Quorum (FTT:2, FTM: RAID-1 ) = 5 Hosts, no stripe FTT =2, FTM = RAID-1, Stripe Width = 1 (VMDK) D (RAID-1) R D D (data component) (data component) (data component) 3 data components = 3 votes (witness component) 1 W W 1 2 witness components = 2 votes 39

40 Votes Re-assigned / Re-balanced as stripe width is changed FTT =2, FTM = RAID-1, Stripe Width = 2 1 (VMDK) C1 (components) R0 (RAID-0) R1 2 2 R0 2 C2 1 (RAID-1) (RAID-0) 1 C1 1 C2 1 C1 1C2 (components) (components) R0 (RAID-0) 2 3 W W Assign higher votes to break tie 40

41 Quorum with stripe width =2 C2 Partition - 1 C (2 votes) W Partition 2 proceeds C2 C1 C2 C1 (2 votes) (2 votes) (1 vote) (1 vote) W (VMDK) Availability but no Quorum (Availability) && (Quorum) 41

42 Quorum = True Availability = False

43 It is possible to have Quorum but no Availability Partition - 1 Quorum C1 C1 1 1 (VMDK) R1 R0 R0 R0 C1 (votes) 1 3 W Partition - 2 Quorum C2 C2 C W 43

44 RAID-5

45 RAID 5 protection against 1 host failure (VMDK) R5 1 C0 1 C1 Each component on a separate host Assigned higher vote to break tie C2 1 C3 esxi-01 esxi-02 esxi-03 esxi-04 45

46 RAID 5 protection against 1 host failure (VMDK) 1 C0 1 D1 D2 P1 D3 C1 R5 Each component is divided into data and parity blocks C2 1 C3 esxi-01 esxi-02 esxi-03 esxi-04 46

47 The Life of vsan Component

48 Object States: can be not compliant but accessible Compliance status: Are all replicas good? 2 R0 R1 R0 C1 C2 C1 C2 (VMDK) Operational status: Is Accessible? 3 Accessible implies Liveness (votes) 2 (votes) W esxi-03 (votes) esxi-01 esxi-02 48

49 Object States: can be not compliant but accessible Compliance status: Are all replicas good? R0 R1 R0 C1 C2 C1 C2 (VMDK) Operational status: Is Accessible? 3 Accessible implies Liveness (votes) 2 2 (votes) W esxi-03 (votes) Active = known good Degraded = known bad, rebuild now Absent = known bad, cause not known, repair after 60 mins esxi-01 esxi-02 Stale = Active however needs update 49

50 4 Rs Resync, Rebuild, Repair and Reconfiguration (Host-1) R1 C1.. C4 (VMDK) (Host-4) (components) (blocks) VMDK is divided into components Components comprise of data blocks Each component on different host Each data block of fixed size R1 C1.. C4 (VMDK) (state: active-stale) (resync blocks) Partial Resync Copy data to stale components When a component comes back from being absent R1 C1.. C4 (VMDK) (build out the component) Repair / Reconfigure Build fresh component Full Resync (state: degraded) 50

51 Rebuild Example

52 Begin: All components / elements are in active state (Active) A A C1 C2 (Active) (Active) W (2 votes) (3 votes) (2 votes) Tolerate 1 host failure with RAID-1 A (Active) A C1 (Active) C2 A 53

53 Cluster partitions with unknown cause, components go Absent Partition - 1 (Absent) Absent: Known bad, but cause not known A B C1 A B C2 2 (2 votes) (Active) A Partition 2 W C1 C2 3 2 (3 votes) (2 votes) A A (Active) Cluster partition, cause unknown, do not repair immediately Object is not compliant but accessible 54

54 Partition with both Availability and Quorum proceeds Partition - 1 (Absent) A B C1 vm HA to partition -2, partition-2 has both quorum and availability C2 2 A B A W Partition 2 - proceeds C1 3 2 A A C2 (2 votes) (3 votes) (2 votes) Availability no Quorum Quorum && Availability 55

55 Partition is resolved, component is Resynced (Active-Stale) AS AS A C1 C2 Resync 2 (2 votes) W C1 3 2 A A C2 (3 votes) (2 votes) Active-Stale Component is Resynced Component marked as Active Stale, Object is not compliant 56

56 All components / elements are in active state (Active) A A C1 C2 (Active) (Active) W (2 votes) A (Active) C1 (3 votes) (2 votes) A (Active) C2 A All components are Active Object is compliant and accessible 57

57 Repair Scenarios

58 Absent Components Repair After 60 Min Partition - 1 (Absent) Resync after 60 min A A W C1 C2 C1 C2 2 A Partition 2 : most recent data 3 2 (2 votes) (3 votes) (2 votes) A A 59

59 Degraded Components Repair Immediately Hardware Failure Causes Degraded (Degraded) Known bad, Resync Now D W C1 C2 C1 C2 2 D (2 votes) A 3 2 (3 votes) (2 votes) A A 60

60 Fresh components Resynced From Existing Components (Degraded) D C1 C2 2 (2 votes) A D Resync W C1 C2 3 2 A A R R C1 C2 2 (Reconfiguring Find another host to resync, Resync begins Object state is not-compliant but accessible (Another Host) 61

61 Object is Compliant Again (Degraded) (remove) D D C1 C2 2 (Active) A C1 (Active) C2 A (Active) W (Active) A A A C1 C (Active) Degraded component is marked for deletion 62

62 Rebuild RAID schematics Resync begins C1 (Degraded) R0 (VMDK) R1 R0 R0 C1 C2 C2 C1 W C2 Resync begins 63

63 Rebuild RAID schematics Resync ends C1 R0 (mark for removal) (VMDK) R1 R0 R0 C1 C2 C2 C1 W C2 Resync Ends 64

64 Reconfiguration Changing Storage Policies

65 Reconfiguration Increase FTT =2 to FTT =3 R1 R0 R0 R0 R1 R0 R0 R0 R0

66 Reconfiguration Increase Sripe Width R1 R0 R0 R0 R0 R0 R1 R0 R0 R0 R0

67 Multi-Level Fault Domains

68 Failures to Tolerate (FTT) can be Nested Survive one site failure and one host failure on the other site vsphere vsan Host Racks Sites 69

69 Stretched Cluster deployment with local fault protection RAID-5 Cluster vsphere RAID-1 3 rd site for witness vsan RAID-5 Cluster Prior examples, host is the fault domain 2 Levels of fault domain Site and host Failures to tolerate at each level 5ms RTT, 10GbE 70

70 RAID tree for stretched cluster with local fault protection D1 D2 P1 D3 R5 R1 R5 D1 P1 D2 D3 (Site -1) (Site -2) 71

71 Survive 1 site failure D1 D2 P1 D3 R5 R1 R5 D1 P1 D2 D3 (Site -1) (Site -2) 72

72 Survive 1 site failure and 1 host failure D1 D2 P1 D3 R5 R1 R5 D1 P1 D2 D3 (Site -1) (Site -2) 73

73 Anatomy of write: from site - 1 to site - 2 2a 1 Issue write Update Local Data and Parity R5 D1 D2 R1 P1 D3 R5 2b Send only data across sites Dn R5 Remote Helper Raid Tree (proxy owner) D1 D2 3 Remote side calculates parity. P1 D3 (Site -1) (Site - 2) 74

74 Votes in Stretched Cluster

75 5 Votes per site 3 voting entities for first level Site-1, Site-2 and the witness 1 1 D1 D2 D3 P1 R5 2 R1 5 5 Witness has equal share of votes as the other 2 entities (e.g. sites) 1 R5 D1 D2 W D3 P1 (Site -1) 4 components for second level Total of 5 votes (odd number of votes) (Site -2) 76

76 Witness is assigned same voting rights as the sites 3 voting entities for first level Site-1, Site-2 and the witness D1 D2 D3 P1 5 R5 R1 Witness has equal share of votes as the other 2 entities (e.g. sites) R5 D1 D2 W 5 D3 P1 (Site -1) 4 components for second level Total of 5 votes (odd number of votes) (Site -2) 77

77 I/O Flows

78 Anatomy of a All Flash Write H1 virtual disk vsphere Virtual SAN H2 5 7 H3 Pretty much same as hybrid: VM running on host H1 H1 is owner of virtual disk object Number Of Failures To Tolerate = 1 Object has 2 replicas on H1 and H2 1. Guest OS issues write op to virtual disk 2. Owner clones write op 3. In parallel: sends prepare op to H1 (locally) and H2 4. H1, H2 persist op to Flash (log) 5. H1, H2 ACK prepare op to owner 6. Owner waits for ACK from both prepares and completes I/O 7. Later, owner commits batch of writes

79 All-flash: Destaging Cache to Capacity H1 virtual disk vsphere Virtual SAN H2 hot H3 VMworld 2017 Data from committed writes accumulate on Flash Cache (Write Buffer) From different VMs / virtual disks In all-flash, blocks that are written most often (hot) stay in write cache. In all-flash, blocks that are infrequently accessed (cold) are destaged to flash capacity layer. Content: Not for publication cold

80 Nerd Out With These Key vsan Activities at VMworld Practice with Hands-on-Labs Learn from self-paced and expert led hands on labs vsan Getting Started Workshop (Expert led) VxRail Getting Started (Self paced) Self-Paced lab available online 24x7 VMworld 2017 Visit SDDC Assessment Lounge Discover how to assess if your IT is a good fit for HCI Four Seasons Willow Room/2nd floor Open from 11am 5pm Sun, Mon, and Tue Learn more at Assessing & Sizing in STO1500BU Become a vsan Specialist Earn VMware digital badges to showcase your skills Content: Not for publication New 2017 vsan Specialist Badge Education & Certification Lounge: VM Village Certification Exam Center: Jasmine EFG, Level 3 #HitRefresh on your current data center and discover the possibilities!

81 3 Easy Ways to Learn More about vsan Storage Hub Technical Library StorageHub.vmware.com Reference architectures, off-line demos and more Easy search function And More! VMworld 2017 New vsan Tools vsan Sizer vsan Assessment Hands-On Lab Test drive vsan for free today! Content: Not for publication Live at VMworld Practical learning of vsan, VxRail and more 24x7 availability online for free! 82

82

83