DESY and NAF. Andreas Gellrich, DESY. 8th Belle II Computing Workshop May 2013, Leinsweiler, Germany

Transcription

1 DESY and NAF Andreas Gellrich, DESY 8th Belle II Computing Workshop May 2013, Leinsweiler, Germany

2 Introduction: Contents > Introduction > The DESY Grid Center > DESY Grid NAF2.0 network > Conclusions Andreas Gellrich DESY and NAF 15 May 2013 Page 2

3 Introduction: Hamburg Andreas Gellrich DESY and NAF 15 May 2013 Page 3

4 Introduction: Grid > Grid computing has shown to be a key technology for the success of the WLCG experiments > In the presentation of the first HIGGS results in July 2012 at CERN, the Grid was explicitly mentioned! > Deployed and operated in the context of the EU projects EGEE/EGI and OSG, the Grid provides computing resources for (many) other VOs e-science > Past experiments such as CDF (Tevatron), H1, HERMES, ZEUS (HERA) as well as future ones (BELLE, ILC) are using the world-wide Grid infrastructure > All VOs benefit from the global approach of the Grid, incl. the seamless access to data world-wide > HEP is data-oriented > Jobs are transient data are persistent Andreas Gellrich DESY and NAF 15 May 2013 Page 4

5 DESY Grid Center: History > DESY has a long tradition in HEP computing (serving experiments on site) > 2003: First look into Grid computing as key technology to access resources > 2004: LCG_2-1 Grid infrastructure H1 and ZEUS, IceCube, ILC, ILDG > 2004: EU-Project EGEE(2/3) / DECH > 2004: Tier-2 for ATLAS, CMS, LHCb > 2005: Foundation of national D-Grid (DGI(2) and HEPCG) > 2007: National Analysis Facility (NAF) (ATLAS, CMS, LHCb, ILC) > 2010: Start of EGI / NGI_DE and EMI (dcache) > 2011: DESY is world-wide biggest Tier-2 center for CMS > 2012: VO 'belle' added > 2013: BELLE@NAF, BELLE II MC campaign, BELLE II data challenge Andreas Gellrich DESY and NAF 15 May 2013 Page 5

6 DESY Grid Center: Grid + NAF > The Grid infrastructure reflects DESY's manifold scientific programme: DESY is the home of 10 VOs (6 global), incl. non-hep (WLCG-)Tier-2 for ATLAS, CMS, and LHCb in Germany (Tier-1: GridKa) Tier-0/1 for HERA; ILC VOs incl. testbeams (DESY, FNAL, CERN); CTA Tape back-end for non-tier-2 VOs > One complete generic Grid infrastructure for all VOs Federated resources w/ opportunistic usage ( everybody profits ) All major Grid core services Currently, roughly 2/3 of the resources are currently used by the Tier-2 VOs > Grid is complemented by the National Analysis Facility (NAF) [size ~1 Tier-2] Andreas Gellrich DESY and NAF 15 May 2013 Page 6

7 DESY Grid Center: DESY-HH > Grid Computing Resources: (CE) [SL5.8] moving to [SL6.4] in summer grid-cr2/3/4/5/6.desy.de hosts Total: 57 khepspec CVMFS > (Disk) Grid Storage Resources: (SE) (dcache) dcache-se-atlas.desy.de ATLAS: 2400 TB dcache-se-cms.desy.de CMS: 3200 TB dcache-se-desy.desy.de others: 700 TB Total: 6300 TB > Network LHCOPN/LHCONE: 20 Gbps [DESY,KIT, GSI, RWTH Aachen, U Wuppertal] X-WIN (DFN): 2x 2 Gbps 2x 5 Gbps (2nd half 2013) Andreas Gellrich DESY and NAF 15 May 2013 Page 7

8 DESY Grid Center: Jobs at DESY-HH (1 year) Running Jobs Waiting Jobs Andreas Gellrich DESY and NAF 15 May 2013 Page 8

9 DESY Grid Center: National Analysis Facility (NAF) > Grid is a global batch system, optimized for massive computing > Final data analysis needs: data! central registry Interactive login to WGS with AFS Batch resources fast(er) response dedicated resources (shares) scratch space, e.g. for n-tupels tools, X > NAF project started 2007 within HGF Successfully used by ATLAS, CMS, LHCB, ILC > NAF2.0 migration in 2013 includes BELLE Andreas Gellrich DESY and NAF 15 May 2013 Page 9

10 DESY Grid Center: National Analysis Facility (NAF) gsissh qsub Interactive Local Batch Proof NAF Grid grid-submit Grid Cluster I n t e r c o n n e c t NAF AFS Parallel Cluster FS AFS/Kerberos scp Dedicated space DESY Grid dcache Storage Grid-ftp, SRM Andreas Gellrich DESY and NAF 15 May 2013 Page 10

11 DESY Grid Center: NAF usage by VO s Andreas Gellrich DESY and NAF 15 May 2013 Page 11

12 DESY Grid Center: NAF usage by VO s Andreas Gellrich DESY and NAF 15 May 2013 Page 12

13 DESY: Grid > The DESY Grid Infrastructure supports the VO 'belle' > A considerable contribution was made to the BELLE2 MC campaign parallel to the normal Tier-2 operations > BELLE2 computing resource requirements for Germany are ~10% of the resources at DESY-HH > Grid services such as WMS (Workload management Services) to submit jobs globally (!) are available at DESY for the VO 'belle' > Storage resources were recently added (currently 300TB) > Support is in place via NGI_DE (National Grid) of EGI (European Grid Infrastructure) via GGUS Andreas Gellrich DESY and NAF 15 May 2013 Page 13

14 DESY: Jobs at DESY-HH (3 month) Running Jobs Waiting Jobs Andreas Gellrich DESY and NAF 15 May 2013 Page 14

15 DESY: DESY-HH usage (3 month) Total: 20 M cpu-hours in 2013 BELLE: 72 k cpu-hours in 2013 Andreas Gellrich DESY and NAF 15 May 2013 Page 15

16 DESY: Grid storage resources DESY total space: 6.7PB DESY total Andreas Gellrich DESY and NAF 15 May 2013 Page 16

17 DESY: Grid storage resources BELLE total space at DESY SE: 300TB Andreas Gellrich DESY and NAF 15 May 2013 Page 17

18 DESY: Data transfer (all-mdst) > DESY started physics studies with BELLE I data on NAF2.0 > all-mdst data should come to DESY (~400TB) > intense discussion with KEK BELLE and CC people > Big differences among WGS in access to data locally > data are now transferred from 'ccw20.cc.kek.jp' > multi-stream much slower than multi-session (WAN or local effect) > 3-4 TB/d with 10 parallel 'lcg-cr' (gridftp) > total of 85TB copied > completed exp Andreas Gellrich DESY and NAF 15 May 2013 Page 18

19 DESY: BELLE I data transfer Andreas Gellrich DESY and NAF 15 May 2013 Page 19

20 DESY: BELLE I all-mdst Andreas Gellrich DESY and NAF 15 May 2013 Page 20

21 DESY: perfsonar: DESY KEK Andreas Gellrich DESY and NAF 15 May 2013 Page 21

22 DESY: perfsonar: DESY GridKa Andreas Gellrich DESY and NAF 15 May 2013 Page 22

23 DESY: NAF2.0 > National Analysis Facility (NAF2.0) ready for BELLE2 > complement to the batch-oriented globally accessible Grid > optimized for interactive analysis > Interactive login to work group servers with AFS > access to the all-mdst data via NFS4.1 (pnfs) > fast scratch space (Sonas) > batch-system with short turn-arounds > provides comparable amount of resources (totals up to 5000 cores) > Belletools installed > registry by Grid certificate to obtain account (coming soon) Andreas Gellrich DESY and NAF 15 May 2013 Page 23

24 DESY Grid Center: NAF2.0 ssh DESY People only ssh Bastion Pub.ifh.de DESY Desktop Private WGS Machine Control ssh Fast network, high BW, low latency Site NN Storage: dcache, Lustre, Sonas, AFS, NFS,... qsub NAF-VO1 WGS ssh NAF-VO2 WGS DESY and external people NX -like ssh NAF-X Remote Desktop Site NN General purpose Batch farm Andreas Gellrich DESY and NAF 15 May 2013 Page 24 DESY internal machines

25 Conclusions > The DESY Grid Center reflects the scientific computing at DESY > One (generic) Grid infrastructure is run for all VOs, federating resources with opportunistic usage > The NAF complements the Grid for interactive data analysis > The VO 'belle' is supported at DESY-HH, incl. Grid services > Considerable resources can be used O(500 slots) > CVMFS is in place for SL6 > The NAF2.0 offers resources for BELLE(2) data analysis > The NAF2.0 is open for more users (soon); registry by Grid certificate > Transferring data from KEK is a challenge and needs further considerations > Do we need BELLE2one (like LHCone) and joint efforts when copying Andreas Gellrich DESY and NAF 15 May 2013 Page 25