Code-Aster parallel performance benchmark: a comparison with Abaqus

Transcription

1 Code-Aster parallel performance benchmark: a comparison with Abaqus J. Cugnoni, LMAF EPFL, Switzerland, 7-05, joel.cugnoni@epfl.ch based on the models developed by D. Comment during his Master Georg Fisher AG Introduction This document aims at presenting an overview of the performance of Code-Aster in parallel computation running on a High Performance Computing (HPC) infrastructure. The performance of Code-Aster.7 (stable version at the date of writing) will be compared to the commercial FE solver Abaqus v6.3 running on the same HPC cluster. To compare with a commercial solver such as Abaqus in which all optimization solutions are typically used, particular attention has been devoted to compile Code-Aster with the latest Intel compiler suite (Intel C/C++/Fortran compiler 5.0. with Intel MKL Math libraries). Moreover, to benefit from the parallelism, Code-Aster was compiled with MPI parallelism (linked to Intel MPI library) for both MUMPS and PETSC solvers. The compilation script for that particular installation is given in appendix. These benchmarks have been executed on EPFL HPC infrastructure using between to CPUs on a single node in order to get an unbiased estimation of the performance without effects of interconnects (regular 0Gb Ethernet for this cluster). Each job has been monitored to make sure that no memory swap or other concurrent processes where running during the benchmark. This work is carried out in a pure Academic research context, without support from industry, with the goal of evaluating the performance of Code-Aster for research applications at LMAF/EPFL. Hardware & operating system Code-Aster was compiled on the so-called Caster HPC cluster at EPFL ( This cluster is configured with Redhat 6 x6_6 and uses the SLURM job scheduler (with hwloc) to run mostly serial jobs in which only one node is used for the calculation. The cluster consists of 50 nodes of dual Xeon 650v3 servers, corresponding to x CPUs and 6Gb DDR RAM per node. The interconnect is 0Gb Ethernet and is used only for user data storage on NFS. A local high speed scratch disk is used for temporary file storage. The benchmark is executed on a single node through SLURM scheduler and a reserved memory of 55 Gb is requested to avoid any swap or concurrent job issues. /

2 Code-Aster.7 is compiled with Intel 5.0. compilers, MKL and Intel MPI libraries (including Intel Scalapack functions) directly on the cluster using regular -O3 optimization flags(no specific optimization flag used for those machines). Abaqus v6.3 was already installed on the cluster but as Platform MPI library was found incompatible with the current version of SLURM only thread level parallelism was used for Abaqus. It should be however noted that Abaqus direct solver with thread level parallelism performs equally well as with MPI when running on a single machine (same overall solver algorithms, with parallelism of both element calculations and linear system solution). A very important point to note for performance comparison is that the latest generation of Intel Xeon processors, such as the Xeon 650v3 used here, are using very aggressive frequency scalings to maximize performance while remaining inside their maximum thermal dissipation envelope. This means that a single CPU job running on an isolated node will most probably run at the max turbo frequency of the CPU (3GHz for the 650v3) while a CPU job will run at a frequency closer or lower than the base CPU clock (.3Ghz here). This means that comparing the simulation times obtained with different number of CPUs does not entirely reflect the scaling of the algorithm but it gives a lower estimate. Benchmark case: simulation of an elasto-plastic tensile test The benchmark test case consists in a simulation of the elasto-plastic response of a cylindrical tensile test specimen. These test cases have been kindly provided by Georg Fisher AG and result from the master project of D. Comment (0). The specimen geometry correspond to German standard DIN 505: with a length of 60mm and min/max diameters of 6 and 0mm respectively. Figure : full specimen geometry (*courtesy of D. Comment) To account for symmetries, the FE model corresponds only to /th of the full specimen and uses thus several symmetry conditions (I.e zero normal displacement) which add a significant number of Lagrange multipliers to the system. The mesh has been generated in Abaqus CAE using the default tetrahedral meshing algorithm and uses 0 node quadratic tetrahedral elements (C3D0). Two uniform mesh sizes have been considered: 0. and 0.35 mm. The Abaqus model was exported as flat Input format (INP) and was converted to Code-Aster format using Calculix CGX preprocessor. Thus the FE meshes used in both solvers are strictly identical. Loading is simulated by imposing a uniform longitudinal displacement ramp on the loading face up to.75 mm corresponding to up to 6% eng. /

3 strain in the gage length of the specimen. An overview of the mesh and boundary conditions is shown in Fig.. Mesh statistics Figure : FE mesh of the problem and boundary conditions (*) Case A: small mesh Uniform mesh size = 0.35 mm 39'096 quadratic tetrahedral elements 6'976 nodes '9 DOFS in Abaqus * 5' DOFS in Aster * Case B: larger mesh Uniform mesh size = 0.0 mm '609 quadratic tetrahedral elements 93' nodes 76'363 DOFS in Abaqus * 977'365 DOFS in Aster * (*incuding Lagrange) The material is modeled as isotropic elasto-plastic with Von-Mises isotropic hardening implemented in a tabular form (TRACTION & VMIS_ISOT_TRAC in Code-Aster). Large deformation analysis is performed in both Abaqus (NLGEOM active: incremental lagrangian formulation with logarithmic strain) and Code-Aster (Simo-Miehe large strain formulation). Loading is performed over an arbitrary time of s with a minimum time step of 0., leading to a minimum of 5 increments during loading. The default solver options are used in Abaqus (adaptive time stepping, default direct solver and Newton algorithm). For Code-Aster, the solution is performed using the Newton algorithm with updated tangent stiffness matrix used for prediction and correction (REAC_ITER=, PREDICTION='TANGENTE', MATRICE='TANGENTE') with a maximum of 0 iterations before time step cut back. The default automatic time stepping strategy of Aster is also used, but it has been verified that in Aster these simulations do not require subdivisions of time steps. In Aster, three different linear solvers are used: 3/

4 a) MULT_FRONT, a standard multifrontal solver with loop level parallelism (through OpenMP). b) MUMPS, a high performance parallel direct solver using MPI parallelism. A distributed, in-core, matrix storage is used here. c) PETSC, a high performance iterative solver (conjugate gradient type) with a single precision factored matrix (LDLT_SP, using MUMPS) as pre-conditionner. In the present case, the preconditionner is computed at each Newton iteration, but a further performance boost might be achieved by choosing a less frequent update strategy. It should be noted that in case a), Code-Aster does not benefit from any other type of parallelism, for example, in the element calculations and thus a relatively limited scaling is to be expected for MULT_FRONT solver. On the contrary, when using MUMPS or PETSC, Code-Aster runs in domain decomposition mode and most of the time consuming tasks are performed in parallel, including element calculations. Thus solvers b) & c) are expected to provide much better scaling when used on many CPUs. It should be noted that the default solver of Abaqus and MUMPS are both direct solvers and thus should be fairly comparable in terms of robustness. On the other hand, PETSC with LDLT_SP is using a potentially more efficient iterative solution technique which may however fail in certain badly conditioned systems (and also is incompatible with contacts at the moment). In all cases, the default convergence tolerances have been used. In terms of output, all solvers have been verified to give identical results, so accuracy will not be discussed in the present document. Performance results The solver performance is simply measured by the wall clock time spent for the execution of the full job. The wall clock time is obtained directly from the message files of Code-Aster and Abaqus. In should be noted that Abaqus does not compute all post-processed variables during the problem resolution. On the contrary Code-Aster computes all post processing variables at the end of the job and thus might spend more time. To counter balance this, the time spent in the Newton solver step is also extracted for Code-Aster and compared with Abaqus. Finally, the solution times for to CPUs are presented in the tables here after: Case A: small mesh ~00kDOFs Total run time = total elapsed time for the run ( in s) CPUs Abaqus Aster MUMPS Aster PETSC Aster MULT_FRONT /

5 Solver time = total elapsed time for the solver only ( in s) Aster MUMPS Aster PETSC Aster MULT_FRONT Case B: larger mesh ~900kDOFs Total run time = total elapsed time for the run ( in s) CPU Abaqus Aster MUMPS Aster PETSC Solver time = total elapsed time for the solver only ( in s) Aster MUMPS Aster PETSC Abaqus Aster MUMPS Aster PETSC Aster MULT_FRONT Figure 3: Total solution time for small case (A) as a function of number of CPUs ( to ) 5/

6 Abaqus Aster MUMPS Aster PETSC Figure : Total simulation time for large model (B) as a function of number of CPUs ( to ) Result overview A first look at the results in Fig 3 & shows that the performance of Code-Aster depends strongly on the chosen solver but is overall in the same range as that of an highly optimized commercial solver like Abaqus. As expected the MULT_FRONT solver does not show interesting parallel performance and is thus recommended for HPC use. On the contrary, MUMPS and PETSC / LDLT_SP lead to a relatively good scaling. Thus, for large scale computation, it is strongly recommended to perform a custom installation of Code-Aster with MPI versions of MUMPS and PETSC solvers. Very interestingly, the best performance for large jobs is achieved using Code-Aster with PETSC/LDLT_SP solver on CPUs in which Code-Aster is actually 7% faster than Abaqus on CPUs. Overall, in large elasto-plastic problems using relatively homogeneous 3D continuum elements, the iterative PETSC solver appears much faster than any of the direct solvers (Abaqus or Code-Aster MUMPS). However, it should be noted that iterative solvers are more sensitive to matrix conditioning and are less robust than direct Cholesky type solvers. Moreover, due to its nature, the iterative PETSC solver in Code-Aster is not appropriate for contact simulation which further limits its use. Despite that, when the problem allows, it is strongly recommended to use PETSC solver with the LDLT_SP preconditionner. It should also be noted that PETSC solver on a single CPU is actually faster than the direct solver of Abaqus on CPUs in the case of large jobs. PETSC appears also much more appropriate for even larger simulations as it scales very well with the increase of model size. For the two models A & B considered here, which correspond to an increase in model size of a factor of.5, the different solvers show a respective increase of simulation time by a factor of 9., 0.7 and 6.9 for Abaqus, Aster/MUMPS and 6/

7 Aster/PETSC (power law exponents of.5,.57 and. resp). If we limit our comparison to direct solvers, the performance of Aster/MUMPS vs Abaqus remains relatively comparable for jobs using to CPUs, with a performance advantage for Abaqus in the order of 30 to 50%. However, as the number of CPUs increases, the performance of MUMPS appear to stagnate faster than the direct solver of Abaqus and does not compete so well here. It should however be noted that the type and performance of the local scratch disk and the time spent for local interconnect remains difficult to estimate and that no machine specific optimization was used for the installation of Code-Aster. Looking at the tradeoff of hardware cost vs licence cost, we can note that the performance of Aster/MUMPS solver with CPUs is roughly equivalent to Abaqus with CPUs. Scaling and comparative analysis To compare those results, we adopt two different points of view. Firstly, the scaling of each solver is evaluated on its own and discussed. Secondly, the performance of the different Code-Aster solvers is compared with respect to Abaqus considering as reference either the solution time using a single CPU or the same amount of CPUs. In the following, the relative performance R is evaluated as : R (solver, ncpus, case) = T_reference / T(solver, ncpus, case) where T_reference represents the reference solution time used as a basis for comparison and T(solver,ncpus,case) is the (total) solution time corresponding to a given solver, number of cpus and simulation case (model size A or B). Solver parallel scaling analysis In this comparison, we want to evaluate how a given solver scales when increasing the number of CPUs. Thus the reference for comparison is the solution time using a single CPU for the same solver and case Abaqus Aster MUMPS Aster PETSC Aster MULT_FRONT 7/

8 Figure 5: scaling of performance (T_cpu/T_ncpus) for small case A (00k DOFs) Abaqus Aster MUMPS Aster PETSC Figure 6: scaling of performance (T_cpu/T_ncpus) for large case B (900k DOFs) From the graphs in Fig 5 & 6, we note that Aster/MUMPS shows a good scaling of up to 5.3 for CPUs in large jobs, but Abaqus solver remains more efficient here with a scaling of up to 9.6 for CPUs. Although very fast in absolute value, the scaling of PETSC/LDLT_SP solver seem to stagnate after to CPUs for models smaller than M DOFs, which also suggests that PETSC is a particularly appropriate solver for very large jobs. In terms of performance vs cluster resource allocation, the optimal number of CPUs can be estimated as follows (considering a limit of CPUs and a single machine): Solver Small jobs (00kDofs) Medium jobs (<MDofs) Large jobs (> MDofs) Abaqus to to (*) Aster / MUMPS (*) Aster / PETSC to max to to (*) (* if within memory limits to run in core ) Aster vs Abaqus comparison To evaluate the advantages / disadvantages of using Code-Aster from a pure performance perspective (no cost comparison here), the performance results of Aster are compared with respect to their equivalent simulations in Abaqus using the same number of CPUs. In the figures below, a time ratio below 00% corresponds to a faster solution time wrt to Abaqus while a value larger than 00% corresponds to a slower resolution. /

9 350% 300% 50% 00% 50% 00% 50% 0% Abaqus Aster MUMPS Aster PETSC Figure 7: simulation time ratio with respect to Abaqus 6.3 (same number of CPUs), small model A 300% 50% 00% 50% 00% 50% 0% Abaqus Aster MUMPS Aster PETSC Figure : simulation time ratio with respect to Abaqus 6.3 (same number of CPUs), larger model B The results show clearly the same trend as discussed previously: when PETSC/LDLT_SP solver can be used in Code-Aster, the Code-Aster is indeed much faster than Abaqus, with solution time down to less than 5% of Abaqus. However, when MUMPS has to be used, a slower solution in Code-Aster is to be expected, in the order of.5 to. times longer than in Abaqus for to CPUs. For the models considered here, using MUMPS with CPUs does not seem so advantageous. As it is very often the case in commercial use of FEA, the actual limitation for computation in HPC is related to the number (and thus cost) of licences. Although a bit artificial, we will considering now a second comparison scenario in which the number of available licence limits the use of Abaqus with only CPU per job and we will highlight the relative performance of the different solvers when 9/

10 compared to an Abaqus job running on CPU only Abaqus Aster MUMPS Aster PETSC Figure 9: relative performance wrt Abaqus CPU, small model A Abaqus Aster MUMPS Aster PETSC Figure 0: relative performance (higher the better) wrt Abaqus CPU, larger model B According to Fig 9 & 0, it is very clear that whenever Aster/PETSC solver can be used a very significant gain of simulation time (up to 3 times faster than single CPU Abaqus job) can be achieved even with a limited number of CPUs (0x speed up for CPUs already). Concerning MUMPS, despite its lower intrinsic performance and scaling wrt Abaqus, a benefit in term of simulation time is still achievable wrt to a single CPU Abaqus job when running it on a large number of CPUs. If the resources allow for typically CPUs per Abaqus jobs, then Aster/MUMPS solver provides roughly equivalent simulation times using CPUs but without licencing costs. If the number of licences is not very restrictive, which is more the case of academic research use, the use of Aster/MUMPS solver over Abaqus is not justified from a pure performance perspective. The advantage of using Code-Aster in this case is mostly to remain free of implementing (and checking) whatever 0/

11 algorithm / model formulation that might be of interest. However, using Aster/PETSC for very large jobs, such as RVE-based homogenization for example, seems a very promising option, especially if we consider that alternative (ie even more efficient) preconditionner updating strategies could be used for even larger speed ups. Intrinsic solver performance Finally, the intrinsic performance of the solver in terms of time per Newton iteration is estimated (within the limits imposed by the CPU frequency scaling effects mentionned previously). In the present case, each Newton iteration corresponds to the assembly and resolution of a full linear system and thus represents well the combination of the core performance of the linear solvers and the efficiency of the element / constitutive law calculations. The number of Newton increments and iterations are summarized here after: Solver & Model Number of increments Number of iterations Abaqus, Small Case A 7 3 Aster, Small Case A (*) 5 5 Abaqus, Large Case B 35 Aster, Large Case A (*) 5 5 (* for all solvers ) In the present benchmark, Abaqus performed a few minor time step cut backs leading to a maximum of increments and a total of 35 Newton iterations, while Code-Aster took only 5 increments (minimum required by minimum time step) corresponding to a total of 5 Newton iterations for all solver used Abaqus Aster MUMPS Aster PETSC Aster MULT_FRONT Figure : time per Newton iteration (in s), small model A /

12 Abaqus Aster MUMPS Aster PETSC Figure : time per Newton iteration (in s), larger model B As mentioned previously, it is clear from those results that, using the default settings and without specific machine optimization, the MUMPS solver used in Aster is not exactly on par with Abaqus implementation as it is spending about x more time per iteration. However, it is important to note that many options exist in MUMPS and none of them have been tuned in that case. Moreover, machine specific compiler optimization options or better OS configuration or execution options might improve those figures. Again, here it is very clear that the PETSC iterative solver with LDLT_SP (single precision MUMPS factorization) is a much more efficient than all the direct solvers considered here (Abaqus & Aster MUMPS/MULT_FRONT). Conclusion Even though this study, as any benchmark, is obviously only representative of the test case considered, very value informations can be drawn considering the performance of Code-Aster parallel solvers. First of all, among all solvers, PETSC with LDLT_SP preconditionner is the most efficient option and provides significant performance speed up even with respect to a very well optimized commercial solver such as Abaqus. However, PETSC/LDLT_SP iterative solver approach limits its use to well conditionned problems, without contacts, for which the single precision factored matrix can be computed and is a good approximation of the real inverse. It is thus ideal in large elasto-plastic 3D continuum problems without localization and instabilities such as the problems considered in this benchmark. Other precondionner could also be used, if LDLT_SP preconditionner fails but these options have not been considered here. Despite showing slower performance in absolute values, Code-Aster / MUMPS option remains competitive with Abaqus if licence usage limits the number of CPUs used. In the present case, which represents a non optimized situation using the default options of the solvers, the MUMPS solver is about x slower than Abaqus and its scaling is not as good. It is however expected that turning off some /

13 matrix pre/post processing options or using another profiler / domain decomposition algorithm could improve the performance of MUMPS. Within the limitations mentionned previously, this study has shown that the performance of Code-Aster.7 (stable version), compiled with MUMPS and PETSC support with Intel tools and math/mpi libraries, shows excellent parallel computation performance that are overall comparable to Abaqus which is considered one of the best performing non-linear parallel FEA solver available today. Appendix In the attached archive AsterAbaqusBenchmark.tar.gz, the following informations can be found (folder structure): / (root of archive) - compile_intel_mpi.sh : fully automatic Code-Aster compilation script with all options - PerfHPC_c.ods: LibreOffice spreadsheet with all results (not perfectly clean... sorry ) - HPC_Benchmark-CodeAster-Abaqus-v..odt &.pdf: the present report - IntelVars.sh : script to load environment for the Intel compilers & libraries / Benchmark / /Small_Model_A_035mm/ /Abaqus/ /Aster/ inp: Abaqus input file (mesh + materials + loading etc..) - ABQ*.run files: SLURM job scripts, contains the job execution command lines and memory/ cpu allocation limits, naming scheme: ABQ.run is for CPUs -.dat,.msg,.sta files: message files corresponding to pre-processor, solver and iteration stats med: MED mesh file - testmumps.comm, testmultfront.comm. TestPETSC.comm: Code-aster command files, including loading, materials and solver options - test*.export: ASTK profiles for execution. Naming scheme: testmumps.export is for MUMPS job using CPUs - test*.run: SLURM job scripts corresponding to each job, same naming as export. - test*.resu, test*.mess: message files containing all timing informations. 3/

14 /Larger_Model_B_00mm/ /Abaqus/ - 00.inp: Abaqus input file (mesh + materials + loading etc..) - for the rest, same as above /Aster/ - 00.med: MED mesh file - for the rest, same as above Revision table: v.0: Initial Draft, JC v.: final Draft, submitted for evaluation / feedback to ProNet, /