Facilitating Data Set Transfers for International Researchers and Showcasing a perfsonar-based Traceroute Monitoring Tool

Transcription

1 Facilitating Data Set Transfers for International Researchers and Showcasing a perfsonar-based Traceroute Monitoring Tool Simon Peter Green Technical Specialist SingAREN

2 Introduction Institut Teknologi Bandung (ITB) requested SingAREN to help improve their dataset transfers between different sources Improve current download performance by : storing datasets closer to ITB such as the SingAREN Database Mirror and using Data Transfer Nodes (DTN)

3 Chiba, JP ITB Weather Dataset Dataset size: 104 MB Links Traversed JP -> SG: 10 G SG -> ID via TEIN: 622 Mbps Source Destination Protocol Throughput Chiba University ITB FTP Mbps Chiba University SingAREN Database Mirroring Service SingAREN Database Mirroring Service Performance increase of 136% FTP Mbps ITB FTP Mbps

4 NOAA, US ITB Weather Dataset Dataset size: ~ 200 MB Links Traversed US -> SG: 100 G SG -> ID via TEIN: 622 Mbps Source Destination Protocol Throughput NOAA NCEP ITB FTP 41.8 Mbps NOAA NCEP SingAREN Database Mirroring Service FTP 89.6 Mbps

5 NOAA File Size Comparison Test Source Destination Protocol Dataset Size Throughput NOAA NCEP NOAA NCEP SingAREN Database Mirroring Service SingAREN Database Mirroring Service FTP 6.6GB Mbps FTP ~200 MB 89.6 Mbps Lower throughput for smaller files TCP congestion control specifically TCP slow-start over a long fat pipe

6 NOAA ITB using DTN Nodes (In Progress) Setup test Data Transfer Nodes (DTN) between US and Singapore Perform the following file transfer tests: 1. NOAA NCEP US I2 DTN Test node 2. US I2 DTN Test Node SingAREN DTN Test Node 3. SingAREN DTN Test Node SingAREN DB Mirror Determine best protocol and configuration for optimum performance

7 DTN System Specifications Host: I2 DTN Test Node SingAREN DTN Test Node CPU: Intel(R) Xeon(R) CPU E GHz Intel(R) Xeon(R) CPU E GHz CPU Core(s): 1 8 CPU L3 Cache: 45 M 15 M Memory: 1.8 GB 32 GB NIC Speed: 10 Gbps 10 Gbps

8 DTN Kernel Configuration (TCP Tuning) Host: I2 DTN Test Node SingAREN DTN Test Node net.core.default_qdisc pfifo_fast pfifo_fast net.core.rmem_max net.core.wmem_max net.ipv4.tcp_rmem net.ipv4.tcp_wmem net.ipv4.tcp_congestion_control htcp htcp net.ipv4.tcp_mtu_probing 1 1

9 Network Route During Test (8/17 & 9/17) Round Trip Time: Bandwidth: Bandwidth Delay Product: 200 ms 10 G 250 MB

10 Usual Network Route to Internet2, LA Round Trip Time: Bandwidth: Bandwidth Delay Product: 182 ms 10 Gb/s 228 MB

11 DTN Test Variables 7 GB File Using 3 different protocols: FTP UDT GridFTPv2 With and without fast retransmissions enabled Different MTU sizes at destination node

12 Throughput (Gbps) 4 Throughput Comparison between FTP & FTP + FAST Number of Parallel Streams FTP Untuned src MTU 1500 dest MTU 1500 FTP FAST Untuned src MTU 1500 dest MTU 1500 FTP Tuned src MTU 8900 dest MTU 1500 FTP FAST Tuned src MTU 8900 dest MTU 1500 FTP Tuned src MTU 8900 dest MTU 9000 FTP FAST Tuned src MTU 8900 dest MTU 9000

13 Throughput ((Gbps) 4 Throughput Comparison between UDT & UDT + FAST (7GB File) Number of Parallel Streams UDT Untuned src MTU 1500 dest MTU 1500 UDT FAST Untuned src MTU 1500 dest MTU 1500 UDT Tuned src MTU 8900 dest MTU 1500 UDT FAST Tuned src MTU 8900 dest MTU 1500 UDT Tuned src MTU 8900 dest MTU 9000 UDT FAST Tuned src MTU 8900 dest MTU 9000

14 Throughput (Gbps) 4 Throughput Comparison between GridFTPv2 & GridFTPv2 + FAST (7 GB File) Number of Parallel Streams GridFTPv2 Untuned src MTU 1500 dest MTU 1500 GridFTPv2 FAST Untuned src MTU 1500 dest MTU 1500 GridFTPv2 Tuned src MTU 8900 dest MTU 1500 GridFTPv2 FAST Tuned src MTU 8900 dest MTU 1500 GridFTPv2 Tuned src MTU 8900 dest MTU 9000 GridFTPv2 FAST Tuned src MTU 8900 dest MTU 9000

15 Throughput (Gbps) 4 Throughput Comparison Between Protocols and Parallel Streams (7GB File) Parallel Streams dest MTU 1500 src MTU 1500 dest MTU 1500 src MTU 1500 dest MTU 1500 src MTU 1500 dest MTU 1500 src MTU 1500 dest MTU 1500 src MTU 1500 dest MTU 1500 src MTU 1500 dest MTU 1500 src MTU 8900 dest MTU 1500 src MTU 8900 dest MTU 1500 src MTU 8900 dest MTU 1500 src MTU 8900 dest MTU 1500 src MTU 8900 dest MTU 1500 src MTU 8900 dest MTU 9000 src MTU 8900 dest MTU 9000 src MTU 8900 dest MTU 9000 src MTU 8900 dest MTU 9000 src MTU 8900 dest MTU 9000 Untuned Untuned Untuned Untuned Untuned Untuned Tuned Tuned Tuned Tuned Tuned Tuned Tuned Tuned Tuned Tuned Tuned Tuned FTP FTP FAST GridFTPv2 GridFTPv2 FAST UDT UDT FAST FTP FTP FAST GridFTPv2 GridFTPv2 FAST File Transfer Configuration UDT UDT FAST FTP FTP FAST GridFTPv2 GridFTPv2 FAST src MTU 8900 UDT dest MTU 9000 src MTU 8900 UDT FAST

16 DTN 7 GB File Transfer Conclusion Optimal configuration: FTP Protocol with FAST enabled average throughput of 3.91 Gbps 8 Parallel Streams Jumbo frames enabled on both DTN nodes Performance significantly increases after TCP tuning and enabling jumbo frames on both source and destination nodes.

17 DTN Test Variables 200 MB File Using 3 different protocols: FTP UDT GridFTPv2 With and without fast retransmissions enabled Different MTU sizes at destination node

18 Throughput (Gbps) 0.4 Throughput Comparison between FTP & FTP + FAST (200 MB File) Number of Parallel Streams FTP src MTU 8900 dest MTU 1500 FTP FAST src MTU 8900 dest MTU 1500 FTP src MTU 8900 dest MTU 9000 FTP FAST src MTU 8900 dest MTU 9000

19 Throughput (Gbps) 0.4 Throughput Comparison between UDT & UDT + FAST (200 MB File) Parallel Streams UDT src MTU 8900 dest MTU 1500 UDT FAST src MTU 8900 dest MTU 1500 UDT src MTU 8900 dest MTU 9000 UDT FAST src MTU 8900 dest MTU 9000

20 Throughput (Gbps) Throughput Comparison between GridFTPv2 & GridFTPv2 + FAST (200 MB File) Parallel Streams GridFTPv2 src MTU 8900 dest MTU 1500 GridFTPv2 FAST src MTU 8900 dest MTU 1500 GridFTPv2 src MTU 8900 dest MTU 9000 GridFTPv2 FAST src MTU 8900 dest MTU 9000

21 Throughput (Gbps) 0.4 Throughput Comparison between All Protocols with and without FAST (200 MB File) Parallel Streams FTP src MTU 8900 dest MTU 1500 FTP FAST src MTU 8900 dest MTU 1500 GridFTPv2 src MTU 8900 dest MTU 1500 GridFTPv2 FAST src MTU 8900 dest MTU 1500 UDT src MTU 8900 dest MTU 1500 UDT FAST src MTU 8900 dest MTU 1500 FTP src MTU 8900 dest MTU 9000 FTP FAST src MTU 8900 dest MTU 9000 GridFTPv2 src MTU 8900 dest MTU 9000 GridFTPv2 FAST src MTU 8900 dest MTU 9000 UDT src MTU 8900 dest MTU 9000 UDT FAST src MTU 8900 dest MTU 9000

22 DTN 200 MB File Transfer Conclusion Negligible performance differences between FTP, UDT and GridFTPv2 protocols Significant performance increase when jumbo frames are enabled on both nodes compared to only the source node

23 DTN File Transfer Conclusion Optimum throughput achieved using: Single large file - e.g. 7 GB 8 parallel streams FTP outperformed UDT and GridFTPv2 for the optimum setup FAST slightly improves FTP performance for large file downloads

24 restore restore restore Proposed File Transfer Process NOAA ITB NOAA Internet2 DTN SingAREN DTN File 1 File 2 File 3 File 4 File 5 File 6 File 7 File 8 File 9 archive archive archive Archive 1 Archive 2 Archive 3 Archive 1 Archive 2 Archive 3 SingAREN Database Mirror File 1 File 2 File 3 File 4 File 5 File 6 File 7 File 8 File 9 ITB

25 Proposed File Transfer Process Details Archive 154 files in 22 file chunks Creates 7 equal archives - ~4.5 GB/archive Total Size: ~31.5 GB

26 Time Taken (mins) Total Time in Minutes to Transfer 154 files (30.8 GB total) from NOAA NCEP, US to ITB, ID NOAA -> ITB (Average) NOAA -> DB Mirror -> ITB NOAA -> US DTN -> SG DTN -> DB Mirror -> ITB Method

27 Time Taken (mins) Time Taken in Minutes to Perform Each Segment within the Proposed DTN Transfer Process NOAA -> US DTN Archive - US DTN US DTN -> SG DTN SG DTN -> DB Mirror Segments File Restore - DB Mirror DB Mirror -> ITB

28 DTN Transfer Process Test Summary Fastest method via DTNs Slowest segments: Approx. 52% of time taken on transfer from NOAA to US DTN - (~21 minutes) Approx. 25% of time taken on transfer from DB Mirror to ITB (~11 minutes) Limited by a 622 Mbps link Singapore to Indonesia via TEIN

29 Acknowledgement Internet 2 Team for providing a US DTN node to test with Dale Finkelson John Hicks Jeff Hagley A*STAR/SingAREN Team looking after the DB Mirror Service Yeo Chang Lim Kenny Hoi Tommy Tan

30 Traceroute Analysis Tool

31 Route Analysis Tool using PerfSONAR Data Tool Written in Python v3 Tested on version and Uses PerfSONAR v4 Esmond archive API to retrieve traceroute/tracepath data Produces results for all recent traceroute tests and stores said results as HTML pages alerts when route changes occur since last script execution

32 Traceroute Matrix

33 Force Graph using D3.js

34

35 Traceroute Matrix

36

37 Alerts on Route Change

38 Demo

39 Q & A

40 Thank you!