Ensuring Deadlock-Freedom in Low-Diameter InfiniBand Networks

Transcription

1 Timo Schneider, Otto Bibartiu, Torsten Hoefler Ensuring Deadlock-Freedom in Low-Diameter InfiniBand Networks

2 InfiniBand Networks Host Channel Adapter (HCA) Switch Channel Input Port Switches and HCAs, connected via unidirectional channels. We model this as a graph

3 InfiniBand Networks

4 InfiniBand Networks

5 InfiniBand Networks

6 InfiniBand Networks

7 InfiniBand Networks

8 InfiniBand Networks

9 InfiniBand Networks

10 InfiniBand Networks

11 InfiniBand Networks

12 InfiniBand Networks

13 InfiniBand Networks

14 InfiniBand Networks

15 InfiniBand Networks

16 InfiniBand Networks Nothing can move now - Deadlock

17 Channel Dependency Graph (CDG)

18 Channel Dependency Graph (CDG)

19 Channel Dependency Graph (CDG)

20 Channel Dependency Graph (CDG)

21 Channel Dependency Graph (CDG)

22 Channel Dependency Graph (CDG)

23 Channel Dependency Graph (CDG)

24 Channel Dependency Graph (CDG)

25 Channel Dependency Graph (CDG)

26 Channel Dependency Graph (CDG)

27 Channel Dependency Graph (CDG)

28 Channel Dependency Graph (CDG)

29 Channel Dependency Graph (CDG)

30 Channel Dependency Graph (CDG)

31 Channel Dependency Graph (CDG)

32 Channel Dependency Graph (CDG) If the CDG is acyclic, no deadlocks can occur. Dally/Seitz, Deadlock-free message routing in milti-processor interconnection networks, 9

33 Mux Demux Dealing with cycles in the CDG Ignore the problem, rely on timeouts / retransmissions (MinHop) Restrict routing such that no cycles can form, i.e. (Up/Down) Use Virtual Lanes (DF-SSSP, LASH) Virtual Lanes (VLs) Physical Link Switch Port Switch Port Each VL has a dedicated set of buffers

34 Virtual Lanes in InfiniBand How does IB implement VLs: The sender sets a Service Level (SL) in the packet header Each switch has a SL-to-VL mapping table which maps (input channel, output channel, SL) to the output VL: denotes physical channel a using VL i The combination of those relation lets us define a virtual routing function Switches cannot access input VL, or change the SL!

35 Utilizing Virtual Lanes - Layering Each path from source to destination uses one VL, each VL forms a layer. If the CDG of each layer is acyclic, there are no cycles Layering moves three edges! We can do better! Which paths to move to minimize #VLs? NP-complete!

36 Utilizing Virtual Lanes VL Hopping IB allows changing the VL within switches needs less resources to break cycles Which paths to move to minimize #VLs? NP-complete?

37 Incrementing the VL If we increment the VL at every hop, the CDG is acyclic: thus thus we can sort D topologically, since < is a total order, therefore D is acyclic. The number of VLs used = number of hops, good for lowdiameter topologies! VL 0 In Out SL VL X 0 S In Out SL VL X X 0 VL 0 VL S VL 0 VL In Out SL VL X VL S For diametertwo networks, incrementing the VL is trivial.

38 Incrementing the VL (diameter > ) Same input/output port, but different VL! S VL 0 VL S VL 0 S VL VL S VL 0 VL VL Leverage the full power of SL-to-VL mapping and use different SLs for different source/destination pairs! Different SL for each pair? 6 SLs in IB, only four leaf switches!

39 Incrementing the VL (diameter > ) S S S S S S6 S

40 Incrementing the VL (diameter > ) S S S S S S6 S

41 Incrementing the VL (diameter > ) S S In Out SL VL S S S S6 S

42 Incrementing the VL (diameter > ) S S In Out SL VL S S S S6 S

43 Incrementing the VL (diameter > ) S S In Out SL VL S S S S6 S SL 0 SL SL SL

44 Incrementing the VL (diameter > ) S S In Out SL VL S S S S6 S SL 0 SL SL SL 0 0

45 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL 0 0

46 In Out SL VL Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL 0 0

47 In Out SL VL Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL 0 0

48 In Out SL VL Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

49 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

50 Incrementing the VL (diameter > ) S S S S S In Out SL VL S6 S SL 0 SL SL SL

51 Incrementing the VL (diameter > ) S S S S S In Out SL VL S6 S SL 0 SL SL SL

52 Incrementing the VL (diameter > ) S S S S S In Out SL VL S6 S SL 0 SL SL SL

53 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

54 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

55 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

56 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

57 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

58 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

59 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

60 Incrementing the VL (diameter > ) S S S S S S6 S SL 0 SL SL SL

61 Results: Slim Fly Topologies (Diameter Two)

62 Results: Slim Fly Topologies (Diameter Two)

63 Results: Dragonfly Topologies (Diameter )

64 Results: Dragonfly Topologies

65 Results: Orthogonal Fat Tree Topologies

66 Age-based arbitration VLs are supposed to be used for QoS and deadlock-avoidance Arbiters can prioritize packets based on VLs not explored in this work, but shown to improve performance by 0% on other networks In our heuristic the VL value corresponds to the age of packets Saving VLs in deadlock-avoidance allows to use them for separation of traffic classes Relevant for datacenters!

67 Conclusions