PHENIX Grid Research Plan – PPDG Year 3
Objectives of the PHENIX grid research efforts in the coming year include the adoption and/or development of tools: 1) supporting overall data management and the large-scale migration of data files between sites; 2) enabling job submission to more than one site; and 3) supporting management of those jobs by users. We intend to accomplish these objectives in large part through collaborative efforts with STAR and US ATLAS, who are developing software toward the same goals.
The approach we plan to take is to investigate existing software, especially that from PPDG sponsored and/or endorsed projects, and whenever reasonable we plan to adopt existing solutions. It may be necessary to adapt existing software to PHENIX-specific environment. PHENIX has neither the desire, nor the manpower, to devise and develop our own independent approach to utilization of grid middleware. As we are co-located at Brookhaven's RHIC Computing Facility/ATLAS Computing Facility, it is natural for us to seek common approaches with STAR and US-ATLAS.
Data management within PHENIX currently handles a few hundred terabytes of raw, processed and simulated data. Current approaches utilize WAN bandwidth between BNL and other sites, including Japan and France, via script-driven bbftp and gridftp. We hope to reduce the need for manpower-intensive oversight of this bulk data movement and allow for better automation and eventual merging with job submission and management tools. Automated registration of new replicas into the PHENIX distributed file catalog is required.
We will investigate tools currently in use by PPDG collaborating institutions; of particular interest are SAM from Fermilab and HRM used at BNL and LBNL. Each existing data management system, however, addresses a set of goals specific to the experiment which designed the system. These may well be somewhat different from our own, and patterns of data usage certainly differ. We will study their specifications and functionality, and evaluate whether these systems can be adopted in whole or in part for PHENIX data management.
There are two aspects to work on job definition for PHENIX. The first is to determine exactly what information is required to fully specify a PHENIX job. Information falls into one of 5 categories:
input data
output data (analysis output or simulated events)
environment, including the PHENIX software libraries, files required implicitly by the PHENIX software (such as magnetic field maps) and analysis/simulation framework
user code, consisting of ROOT macros, compiled code modules, along with control scripts
parameter files
The second task is to determine requirements for a Job Description Language sufficient to fully specify a PHENIX job. This will be used to collaborate with STAR and US-ATLAS to evaluate and extend STAR's prototype JDL to also meet the PHENIX requirements.
We will evaluate the applicability of the STAR job scheduling software to job submission and management for PHENIX. To accomplish this, we will implement the scheduler for PHENIX, modify one or several PHENIX batch job control scripts to produce STAR-style JDL as output, and use the result to test the functionality of the scheduler for PHENIX nodes at the RCF.
Working collaboratively with STAR, we will evaluate the performance, identify desired additional functionality, and determine how best to further develop needed elements. Key areas for PHENIX include the ability to utilize two or more PHENIX sites, usage of the PHENIX replica database ARGO to locate input data and register output data, data movement to final output site specified by the user, and ease of utilization by new users. Possible areas of general applicability where we may ultimately be able to undertake developments together with STAR include implementation of various policies for resource brokering within the scheduler, and the development of a database module and interfaces for information about jobs, status, errors, and data files. While the former is a likely candidate for PPDG year-4 activities, the latter are needed sooner. We are also very interested in a web interface for job submission and management. Such an interface should show job status and perform control functions, such as stopping or resubmitting jobs.
In developing grid job submission and management, the requirements for PHENIX include:
Location of input data is an important ingredient in the selection of sites by the scheduler. It is imperative for any scheduler to query the replica database as part of the site selection process.
Input and output data sets may be large, and are kept in HPSS or local disks. Output data would initially be written to local disk space and later moved to central storage at the RCF. Movement could be done in two stages, either at the end of each individual job, or after collection of output data from a set of jobs. The replica database will be automatically updated at this step.
Users should have a choice of several interfaces for job submission. They may choose to develop master scripts controlling multiple jobs within a pass on a large data set. Alternatively, a GUI to allow new users a web-based job submission is also envisioned.
Job management should support queries about individual jobs, which may be run at different sites.
The system must allow users to override scheduler decisions and select a specific site.
Job management provides access to: job name, user name, job hierarchy position (i.e. control, collection, or individual file processing steps), site selected for the job, submission host name, job start and end timestamp and execution time, input and output files logical and physical names, environment variables at job start; sub-jobs should point to the master job, log and error files.
We envision initial deployment of a prototype job management system in the coming year, with an initial focus on handling PHENIX simulation jobs. After evaluation and an initial development stage, we will work with a small number of PHENIX collaborators and train them to use the system. Their feedback will drive a second development and improvement effort.
As we are co-located at Brookhaven's RHIC Computing Facility/ATLAS Computing Facility, it is natural to seek common approaches with STAR and US-ATLAS. Establishing a collaborative effort with the other experiments will allow PHENIX to use existing PPDG-sponsored products as a starting point. After a period of evaluation and adaptation of existing tools addressing our objectives, we expect to contribute toward joint further development.
Of paramount interest are joint efforts to define an appropriate Job Definition Language that can fully specify data analysis jobs for all three experiments. The other opportunity for close collaboration is in job scheduling. The STAR collaboration has started development of a scheduler, which is currently operating regularly within the RCF. PHENIX computing is carried out on several large sites outside the US (the computing center at RIKEN in Wako, Japan and the IN2P3 center in Lyon) and multiple medium-scale sites inside the U.S. (e.g. RAMData at Stony Brook, Vampire at Vanderbilt University, Los Lobos at University of New Mexico, and several computing facilities at Lawrence Livermore Laboratory). A natural point of collaboration for PHENIX is in extension of the scheduler to be able to query resources, and submit and manage jobs at the PHENIX sites. Of particular interest is the RAMData facility at Stony Brook, which is managed by PHENIX personnel who also will collaborate on the grid developments. We look forward to utilization and extension of the STAR Job Scheduler, and development of job dispatching policy to utilize RAMData for PHENIX jobs. As the same people testing multi-site job dispatching also manage this site, site-specific installations can be easily addressed.