Setup of a multi-os platform based on the Xen hypervisor. An industral case study. Paolo Burgio

Transcription

1 Setup of a multi-os platform based on the Xen hypervisor An industral case study Paolo Burgio paolo.burgio@unimore.it Roberto Cavicchioli Ignacio Sanudo Olmedo Marco Solieri

2 Who are we? High-Performance Real-Time Lab Since 4 profs, 7(+1) post-docs, 4(+2) PhDs, 1 admin + undergrads University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

3 What we will see today An OPEN-NEXT use-case System and problem description Current system benchmarking / profiling Proposed solution & benchmarking Synthetic workloads Real application 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

4 System description

5 The case study Industrial automation system for packaging x86 host pc with Windows for HMI + ARM embedded PC for image processing in prod. line Goal: meet maximum comm. BW (100Mbps) under RT 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

6 Background What is a real-time computing system? Reacts within precise time constraints to events in the environment Deterministic behavior Windows determinism 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

7 Background Real-time classification Hard: missing a deadline is a total system failure. This case study Firm: infrequent deadline misses are tolerable, but may degrade the system's quality of service. The usefulness of a result is zero after its deadline. Soft: the usefulness of a result degrades after its deadline, thereby degrading the system's quality of service University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

8 The target machine CPU: 4 core Intel(R) Core(TM) i7-4770t 2.50GHz Windows 7 Soon, multi-os (Win + Linux) Memory: 32GB Network: x8 I350 Gigabit Network Connection 82574L Gigabit Network Connection Storage: [0:0:0:0] disk ATA Samsung SSD 850 2B6Q /dev/sda (128GB) [1:0:0:0] disk ATA Samsung SSD 850 2B6Q /dev/sdb (128GB) [2:0:0:0] disk ATA ST2000DM001-1ER1 CC25 /dev/sdc (1,8T) 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

9 System profiling

10 Synthetic benchmarks CPU Setup 1. High priority task that calculates a prime number 2. Two low priority tasks that calculate π Network Setup 1. High priority IPERF3 task (Network computation) 2. CPU prime95 interference (CPU computation) Current (production) system Mono-OS (Win) 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

11 Benchmark suites PRIME95 Computes prime number(s) FFT iperf3 Tool for active measurements of the maximum achievable bandwidth on IP networks It supports tuning of various parameters related to timing, buffers and protocols (TCP, UDP, SCTP with IPv4 and IPv6) For each test it reports the bandwidth, loss, and other parameters Three-clause BSD license 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

12 Execution Time (ms) CPU - A problem: OS power mgmt 1, , , CPU Low Latency test (Real-Time task) FASTER WITH INTERFERENCE??? 0, , , , Num test WindowsNonInterference WindowsInterference(U:70%) 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

13 Execution Time (ms) CPU Exec time unpredictability 250, , , CPU Low Latency test (Real-Time task) 200x variance! 100, , , Num test WindowsInterference(U:100%) 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

14 Bandwidth (Mbits/s) Network sensitiveness to CPU interference Iperf Windows NumTest WindowsPrimeInterference WindowsNative 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

15 February 20th, Report #1 Main issues reported Problem with the power management We can not guarantee real-time constraints using Windows CPU computation messes up the (independent) network computation! We must guarantee component isolation Spatial Timing 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

16 Proposed solution virtualization!

17 Multi-application: traditional approaches Application Application Operating system Operating system Determinism Performance...? 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

18 Proposed system configuration Xen Hypervisor New platform generation Mixed Linux/Win SSD1 SSD2 HDD St: 20GB St: 40GB 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

19 Measure the hypervisor overhead Goal: show the (non-)latencies introduced by the hypervisor: Windows Virtual Machine (VM) vs Windows Bare Metal (BM) Linux VM vs Linux BM Benchmark single-os vs multi-os interference Two tests CPU - intensive (FFT) Memory - intensive (R/W) University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

20 No interference (single-os)

21 Execution Time (ms) CPU - Xen overhead vs. BM (Win) FFT Windows FFT Length BEST_BM AVG_BM BEST_VM AVG_VM 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

22 Execution Time (ms) CPU - Xen overhead vs. BM (Linux) FFT Linux FFT Length BEST_BM WORST_BM AVG_BM BEST_VM WORST_VM AVG_VM 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

23 MB/s Mem - Xen overhead vs. BM (Win) Windows R/W MEM Rate Block Size (Kb/s) WritingBM ReadingBM WritingVM ReadingVM 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

24 MB/s Mem - Xen overhead vs. BM (Linux) Linux R/W MEM Rate Block Size (Kb/s) WritingBM ReadingBM WritingVM ReadingVM 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

25 Inter-OS interference

26 CPU - Test outline High priority task running in the Linux PREEMPT_RT partition that calculates a prime number Two low priority tasks that calculate a pi number running in the WINDOWS partition Btw few linux RT extensions 1. PREEMPT_RT patch (not mainline yet) 2. SCHED_DEADLINE is mainline 3. GRUB is (recently) mainline 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

27 Execution Time (ms) CPU - cross-os interference 0,05 0,045 0,04 0,035 0,03 0,025 0,02 0,015 0,01 0,005 0 PREEMPT_RT CPU Latency Test Less than 15% avg variance! Num test PREEMPT_RT WITHOUT INTERFERENCE PREEMPT_RT W/ INTERFERENCE 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

28 Execution Time (ms) Execution Time (ms) CPU - comparison Windows CPU Low Latency test PREEMPT_RT CPU Latency Test 250, ,05 0, , ,04 0, , ,03 0, , ,02 0,015 50, ,01 0,005 0, Titolo asse Num test WindowsInterference(U:100%) PREEMPT_RT WITHOUT INTERFERENCE PREEMPT_RT W/ INTERFERENCE 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

29 Network Test outline Latency (Roundtrip time) test Like a TCP version of "ping": roundtrip network communication PRIME95 as interfering 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

30 Bandwidth(Mb/s) Network OS interference Iperf VM PREEMPT_RT Num Test LinuxPRTNative LinuxPRTWithInterference 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

31 Bandwidth (Mbits/s) Bandwidth(Mb/s) Network - comparison Iperf Windows Iperf VM PREEMPT_RT NumTest Num Test WindowsPrimeInterference WindowsNative LinuxPRTNative LinuxPRTWithInterference 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

32 and what about the power manager?

33 Execution Time (ms) Power management 0,08 0,07 0,06 0,05 0,04 0,03 0,02 0,01 0 PREEMPT_RT CPU Task Num test Power saving LOW FREQUENCY HIGH FREQUENCY High-performance 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

34 Xen-based power mgmt Xen provides a simple way to configure the per-core frequency $ xenpm set-scaling-governor ondemand performance powersave 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

35 Take-away lessons Identified problems Power management still unresolved Mixed Linux/Xen power management, Win? Windows real-time (while Linux has RT- extensions) Work synergystically with Xen? CPU computation interference network computation on shared resource access Xen-level policies 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

36 Real application (OPEN-NEXT use-case)

37 Use-case application RT partition Read a file from SSD and send it multiple time over ETH/TCP-IP. We want to saturate the 1Gbps BW of each of the 8 ETH adapters (8 independent files) Non-RT partition Test 1 HMI-like (graphical) workload. Up to 90% CPU utilization + write on HDD Test 2 Also, transfer data from HDD to SSD, and sent to (dummy) printer Examples Tile 60x60 -> 10k x 10k pixel -> 25MB file per channel; 30 tiles/min minuto; Period: 255 s Tile 1800x3600 -> 29k x 57k pixel -> 384MB file per channel; 12 mt/min al minuto; Period: 1.4 s Very small image -> 200 KB per channel -> 1k images/min; Period: 0.06 s 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

38 Application scheme 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

39 Real-time OS-to-core mapping Partitioning OSs pinned to cores (Core 0 => Xen Dom0; Core 1 => Linux; Core 2,3 => Win) Scheduling OSs not pinned to cores: Xen performs allocation 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

40 Execution time (ms) Test 1 Partitioning approach Printing Test1 Orignal (Win) applicatoin Num test Xen Partitioning Non Interference Xen Partitioning Interference SACMI Non Interference SACMI Interference 99% Mixed RT-Linux + non- RT win 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

41 Execution time (ms) Test 1 Scheduled approach Scheduled Test1 Num test Xen Sched Non Interference Xen Sched Interference SACMI Non Interference SACMI Interference 99% 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

42 Execution time (ms) Test 2 - Partitioning 2500 Copying files Printing Test2 Copying files Copying files Copying files Num test Xen Partitioning2 SACMI University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

43 Execution time (ms) Test 2 - Scheduling 2500 Copying files Scheduled - Test2 Copying files Copying files Copying files Num test Xen Sched2 SACMI University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

44 .even more interference

45 Execution time(ms) Test 3 - Partitioning PRIME Partitioning Test3 Num test Xen Partitioning3 SACMI University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

46 Execution time(ms) Test 3 - Scheduled Scheduled Test3 Num test Xen Sched3 SACMI University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

47 Take-away lessons Xen hypervisor adds no overhead Can be effectively used to increase isolation/predictability for soft-rt systems Open-source, can be modified / further improved Windows uneffective in RT: Linux w/rt extensions provide quite good results Some tuning is beneficial No modifications (yet) to the source code Still, power management is an issue Platform-specific solutions? 2017 University of Modena and Reggio Emilia OPEN-NEXT dissemination event, October 18th,

48 Thank you! Paolo Burgio Roberto Cavicchioli Ignacio Sanudo Olmedo Marco Solieri