MPAS on GPUs Using OpenACC: Portability, Scalability & Performance

Transcription

1 MPAS n GPUs Using OpenACC: Prtability, Scalability & Perfrmance Dr. Raghu Raj Kumar Prject Scientist I & Grup Head Special Technical Prjects (STP) Grup Natinal Center fr Atmspheric Research March 2018

2 Outline Mtivatin & Gals System & Sftware Specs Apprach & Challenges Results Future Plans Questins 2

3 Mtivatin & Gals Mtivatin Shallw water equatin slver & Discrete Galerkin Kernel Shwed prmising results n GPUs Unified cde was pssible- at least fr small scale applicatins Gals Prt MPAS n t GPUs Optimize perfrmance n GPUs N cmprmise n prtability: < 10% f riginal perfrmance Scale MPAS n GPUs 3

4 System Specs NVIDIA s Internal Clusters PSG: Dual scket Haswell (32 cres/nde) with 4 P100/nde, 12 ndes, intra-nde PCIe cnnected, inter-nde FDR PSG: Dual scket Haswell (32 cres/ nde) with 4 V100/nde, 2 ndes, intra-nde PCIe cnnected, inter-nde FDR NCAR s Cheyenne supercmputer Dual scket Bradwell (36 cres/nde), 4,032 ndes IBM s R92 Cluster (Internal) Minsky- Dual scket Pwer8 (20 cres/nde) with 4 P100/nde, 90+ ndes, intra-nde NVLink, inter-nde IB 4

5 Sftware Spec: MPAS Dry Dynamical Cre Sftware MPAS Release (r MPAS 5.2) Intel Cmpiler 17.0, PGI Cmpiler Dry Barclinic Instability Test- N physics, n scalar transprt Dry dynamics test-case prduces barclinic strms frm analytic initial cnditins Split Dynamics: 2 sub-steps, 3 split steps 120 km (40k grid pints, dt=720s), 60 km reslutin (163k grid pints, dt=360s), 30 km reslutin (655k grid pints, dt=180s), 15 km reslutin (2.6m grid pints, dt=60s) Number f levels = 56 Duble precisin (DP) and Single precisin (SP) Simulatin executed fr 16 days, perfrmance shwn fr 1 timestep 5

6 Why did we chse Dycre? MPAS Physics scheme MicrPhysics WSM6(9.62%) Dynamic Cre Bundary Layer YSU(1.55%) Gravity Wave Drag GWDO(0.71%) Radiatin Shrt Wave RRTMG_SW(18.83%) Executin time- Physics: 45-50% DyCre: 50-55% Lines f Cde- Physics: 110,000 DyCre: 10,000 Radiatin Lng Wave RRTMG_LW(16.43%) Flw Diagram by KISTI Cnvectin New Tiedtke(4.19%) 6

7 Dycre: Zm in MPAS km case with Intel Cmpiler running 36 MPI ranks n Intel E5V2697v4 Bradwell nde substep 5% Integratin Setup Mist 1% cefficients 0% imp_cef 1% MPI 8% diagnstics 20% dyn_tend 32% large_step 16% acustic_step 13% small_step 4% 7

8 Apprach Sftware & Architecture Baseline KGen Prting OpenACC Directives KGen Cnfiguratin & Accuracy Optimize Prfile & Analyze Benchmark Testing Verificatin Integrate Prtability Check Cde Refactring

9 Challenges Faced: Using the Right Directives Lwer time fr prting, Reasnable perfrmance Much higher time fr prting, Imprved perfrmance depending n the lp cunt- up t 50% 9

10 Challenges Faced: Using the Right Apprach fr Data subrutine dyn_tend!$acc enter data!$acc data cpy( ) <cde>!$acc exit data end subrutine dyn_tend Create a cpy n Hst and Device simultaneusly. Harder t design, N unnecessary cpies f data between hst and device Lwer time fr prting, Repeated unnecessary data transfer- pr perfrmance subrutine dyn_tend!$acc data present( ) <cde> end subrutine dyn_tend 10

11 Challenges Faced: Using the Right Techniques Understand hw t ptimize Learn the basics f ptimizatin Understand hw GPUs wrk Pr perfrmance snippets Knw when t use glbal, register and shared memry Nvprf is yur best friend! Understand hw GPUs and CPUs wrk Experiment with SIMD friendly lps- cde layut Experiment with GPU s SIMT cde- data layut Learn hw t cmbine the tw! 11

12 Timers Single Nde Perfrmance: MPAS Dry Dycre MPAS GPTL timers reprted in lg files GPU Timing : Has n updates frm device t hst Hst updates maybe needed fr printing values n screen Hst updates maybe needed fr netcdf file utput Dataset Bradwell (Fully Subscribed, OpenMP Enabled, Intel cmpiled, Base cde) P100 with Haswell(1 GPU, PGI cmpiled, OpenACC cde) V100 with Haswell (1 GPU, PGI cmpiled, OpenACC cde) P100 with Pwer8(1 GPU, PGI cmpiled, OpenACC cde) Speedup Bradwell vs P100 Speedup Bradwell vs V Km (40K) 60 Km (163K) SP DP SP DP Taking 40k data pints per nde fr SP, 32.8M grid pints (15 Km & 3 Km Lcally refined grid) need ~800 Vlta GPUs 12

13 Time (secs) Weak Scaling fr MPAS Dry Dycre (SP & DP) n P100 GPU Number f GPUs 40k per nde, SP 40k per nde, DP 163k per nde, SP 163k per nde, DP Time per timestep, 4 GPUs per nde, 1 MPI rank per GPU, Max f 4 MPI ranks per nde, Intrande Affinity fr MPI ranks, Uses OpenMPI, PCIe n NVLink, PGI

14 Inverse Time Strng Scaling fr MPAS Dry Dycre (SP & DP) fr 15 Km (2.6M) n P100 GPU Number f GPUs SP DP Time per timestep, 4 GPUs per nde, 1 MPI rank per GPU, Max f 4 MPI ranks per nde, Intrande Affinity fr MPI ranks, Uses OpenMPI, PCIe n NVLink, PGI

15 Time (secs) Prtability: Perfrmance Cmparisn f Base cde with OpenACC cde n Fully Subscribed Bradwell single nde k SP 40k DP 163k SP 163k DP Dataset Base Cde OpenACC Cde Fr datasets up t 40k per nde, the executin time is identical. Fr 40k & 163k, the variatin is <1% & <4% respectively. 15

16 Future Wrk Imprving MPAS Scalability MVAPICH instead f OpenMPI NVLink systems Mving hal exchange bk-keeping n GPUs MPS: Preliminary results shwed n imprvement Scalar Transprt Currently being integrated Physics Prt: 35% remaining Optimize: 65% remaining Radiatin, Land surface n CPU Develpment Lagged radiatin Adpting Sin library fr faster IO 16

17 Thank yu! Questins? 17