Porting Applications to the Grid Charles Loomis Laboratoire de l Accélérateur Linéaire, Université Paris-Sud 11, Orsay, France Lecture given at the Joint EU-IndiaGrid/CompChem GRID Tutorial on Chemical and Material Science Applications Trieste, 15-18 September 2008 LNS0924004 loomis@lal.in2p3.fr
Abstract Routine, heavy use of the EGEE Grid infrastructure by a broad spectrum of different scientific disciplines is a reality. The Grid infrastructure offers a collaborative environment where scientists can easily share not only their resources but also their data and expertise. Porting scientific applications to a Grid infrastructure can be challenging, but the EGEE project provides support for porting new applications, helping expand and diversify the user community and running applications. Applications from high-energy physics, biology, earth science and other disciplines prove the utility of the Grid infrastructure for scientific research.
Contents 1 Enabling Grids for E-sciencE 59 1.1 EGEE Grid infrastructure.................... 59 1.2 glite middleware......................... 59 1.3 Support services......................... 60 2 Porting applications 60 3 Conclusions 61
Porting Applications to the Grid 59 1 Enabling Grids for E-sciencE A Grid provides uniform access to distributed computational resources, federating resources from different administrative domains or institutions, requiring a coherent user authentication and authorization framework throughout. Scientists benefit from a Grid infrastructure because it permits individuals to perform new, larger, or more precise calculations than they would be able to do alone. The Grid acts as a platform for sharing data and expertise (e.g. code or algorithms). In short, it provides an ideal platform to analyze data, to publish result datasets, and to combine previous results. EGEE 1 is a publicly-funded project that provides a service-oriented Grid infrastructure. The infrastructure serves primarily the European scientific community, although the project has contacts with researchers throughout the world as well contacts with industry. It collaborates with other Grid projects like Open Science Grid (OSG) 2 in the United States, Nordic Data- Grid Facility (NDGF) 3 in Northern Europe and NAREGI 4 in Japan, for example. 1.1 EGEE Grid infrastructure By nearly all measures, the EGEE Grid infrastructure is the largest in the world with 260 sites from 45 countries participating. Around 80000 CPU cores and many petabytes of disk space are accessible via the Grid infrastructure. More than 13000 registered users, from many scientific disciplines, take advantage of these resources. In the first half of 2008, the total CPU utilization on the EGEE Grid infrastructure was approximately equivalent to 28000 CPUs running continuously over that 6-month period. The total CPU utilization has been doubling every 12 to 18 months and will likely continue to do so. 1.2 glite middleware The glite middleware 5 provides the core functionality available on the EGEE Grid infrastructure. Including software developed both by EGEE itself and 1 http://www.eu-egee.org/ 2 http://www.opensciencegrid.org/ 3 http://www.ndgf.org/ 4 http://www.naregi.org/index e.html 5 http://glite.org/
60 C. Loomis by other projects, glite provides a core software stack on which to run scientific applications. This core includes a common security infrastructure, services to access resources, and some high-level services, like meta-schedulers. Real applications, however, usually need to use quite a lot of software on top of glite. Although some of this additional software really is specific to the application, much of it can take the form of generic functionality or services. The EGEE project maintains a list of useful third-party software, providing for instance, encrypted data management, tools for large taskbased calculations, interactive access, etc. 1.3 Support services The EGEE project provides a full set of support services to the user community. Included are help-desk support, administrative support, and porting support (see below). In addition, the project regularly holds training sessions and tutorials for new and existing users. These traditional support services are complemented with annual User Forums that allow users to exchange information concerning their experiences with the Grid. 2 Porting applications Often people come to the Grid with vague ideas about how the Grid can improve their scientific work. For people not experienced with Grid technology it can be extremely difficult to discover the best services and techniques to use when running an application on the Grid. An application support team within EGEE acts as consultants to discuss with scientists their applications and to develop a specific porting plan. This helps people better understand the EGEE Grid services and avoids initial frustration with the significant learning curve associated with Grid technology. The nature of Grid infrastructures makes some applications more difficult to port than others. Those that are easy to port include trivially parallel applications, parameter sweep applications, those that calculate statistical properties from independent calculations, and those doing simple analyses of file-based data. Slightly more difficult to port are lightly-coupled applications, weakly-parallel or small MPI-based applications, workflows (chains of coupled applications), and those using metadata or databases. The most difficult to port are interactive applications, applications requiring their own services, and applications using commercial (or binary-only) executables.
Porting Applications to the Grid 61 There are examples of all of these types of applications running on the infrastructure. Probably more important is to know what applications are not suited to the EGEE Grid infrastructure. Large-scale parallel applications with significant communication between the nodes, typically run on supercomputers, are not well-adapted to the EGEE infrastructure. At the other end of the spectrum, short, one-time analyses are usually better done on a desktop machine, avoiding the sometimes large latencies involved with Grid computations. 3 Conclusions Routine, heavy use of the EGEE Grid infrastructure by a broad spectrum of different scientific disciplines is a reality. The general nature of the EGEE Grid infrastructure accommodates a broad range of different application types. Most importantly, the Grid serves as an ideal collaborative platform allowing scientists to analyze, publish, and combine previous results all within the same environment. EGEE and Grid techology have evolved over the years and will continue to evolve to better meet the needs of the user community by expanding the types of resources available on the Grid, by introducing new services, and by incorporating promising technologies. The hope is that the EGEE projects will be followed by an initiative to guarantee the availability of this critical computational platform as a stable European Grid Infrastructure in the future. Acknowledgements The author gratefully acknowledges the support of the European Commission and of the various national governments participating in the EGEE project and their collaborators within the application activity (NA4) within the project. This work is co-funded by the European Commission through the EGEE-III project, contract number INFSO-RI-222667.