For the Solenoid Tracker At RHIC (STAR) collaboration & The Information Technology Division (ITD)
Grid
Collaboratory Pilot activities & iVDGL expression of interest
Expression of interest in iVDGL participation
Overview
The BNL[1]’s STAR[2] experiment and ITD[3] have teamed together to achieve several important tasks on the path toward consolidating their grid infrastructure and fully deploying a Grid environment by 2005 for the BNL community. However, much is still to be done. In order to complete such an environment, including its operational aspects, it is appropriate for us to explore new approaches to sharing and competitiveness, and to seek new partnerships within the field and with those outside the field. Furthermore, long term planning and comprehensive Grid strategy is an essential component to our success and within this scope, this document is an opening to partnership with iVDGL.
STAR & ITD Grid activities
The STAR Collaboration
The STAR experiment is one of the four Relativistic Heavy Ion Collider (RHIC) experiment at the BNL. Unlike traditional Nuclear Physics experiment, the STAR experiment records each year massive amount of data: up to a PByte of data a year (1012 bytes) is being accumulated representing to date an integrated total size of 3 PB spanning over 3 million files.
The projection for the next RHIC run (also called, Year4 run which will start by the end of 2003), shows and increase by a factor of five on the number of collected events and brings our production time turn around, using traditional methods, to the next order of magnitude: from month to year and within a very active and aggressive physics program, the local available processing resources will be severely challenged for the science to be delivered in a reasonable time scale. The situation will become more and more problematic as our Physics program evolves toward the search for rare probes: the currently decadal plan (next 10 years STAR activities development) clearly described the need for several upgrade phases including a factor of 10 in data taking and throughput by 2007.
Data management (including making available the second pass of production physics summary tape to our remote institutions) and pass0 facility production become problematic. However, the STAR computing strategy has envisioned and planned to cope for those difficulties by engaging itself in Grid activities and the use of middle-ware tools to resolve some of those issues. However, it becomes clear that we must expand and reach other Grid collaboration for us to rapidly converge and consolidate the infrastructure.
STAR is already a member of the Particle Physics Data Grid (PPDG collaboration) and its mission statement includes dissemination of knowledge and Grid awareness at BNL and has engaged in bringing the local experiments and teams together toward the realization joint activities. In this regard, STAR has initiated and carried along many cross experiment (Phenix/RHIC, Atlas), department (ITD) or Grid related projects and activities beyond its limits (J-Lab, NERSC
The Information Technology Division
(ITD)
A major component of ITD’s mission is to
support the scientific research and advanced technology initiatives of the
Laboratory by:
·
Providing
a reliable and secure high speed networking structure for scientific and
business computing.
·
Developing
a scientific computing infrastructure that can support the computing needs that
are common among the scientific departments.
·
Providing
support to scientific programs to help them address their unique computing
problems.
·
Providing
a cost-effective, highly reliable, secure and standard computing infrastructure
for business and administrative communication.
·
Promoting
best practices in the areas of hardware, software, training and support.
ITD’s Scientific Computing
Support Section provides facilities and expertise to enable and improve the
computational science posture of the Laboratory.
- It provides the
Brookhaven Computing Facility for major server systems, including the
major shared enterprise-wide platforms, RHIC[4]/Atlas,
Riken, and CDIC[5].
- It has established two
centers of expertise; for data visualization and grid and application
software development.
- It has specific
responsibilities for fabrication, operation, and software development for
the Riken machines.
- It provides
professional programmer support directly to Laboratory research areas,
such as RHIC.
ITD also collaborates with the RCF, STAR and Atlas on their growing Grid. The ITD's role is to bring the technology and lessons learned from incorporating GRID into the activities of these projects to the other scientific disciplines at BNL and to ensure compliance with DOE regulations.
The STAR & ITD Grid activities
Since the beginning of its formation, the STAR collaboration has been part of the PPDG Collaboratory Pilot. Although financially lightly supported, the progress and activities have been substantial.
Our past year's activities
have been clearly focused on the realization of a production Grid
infrastructure and deployment of a set of tools without disrupting our main
RHIC program that is our primary mission to the DOE, namely, delivering science
to our NSF and/or DOE institutions/university and the scientific community at
large. Our main strategy was therefore oriented toward the introduction of either
additional (low risk) tools or providing to the user stable and robust APIs
shielding our scientists from the fabric details.
A few activities can
illustrate the strategy and path we have taken:
- With the use of the
SRM tools, over 20% of our data is reliably shipped between the BNL
(Brookhaven National Laboratory) and LBL (Lawrence Berkeley Laboratory).
The use of this tool enables us to remove the routine data management
tasks, and instead allocate the limited manpower to the next phase of Grid
tools and needs.
- Largely driven by the
need to prepare our user community to running on the Grid, we have
developed a job submission system (wrapper around Grid middleware) and
engaged into the strong support of interactive analysis tools on the Grid
(interfacing with SRM and making data movement transparent).
- In addition to the
batch submission, the STAR collaboration has the unique capability of a
interactive user analysis framework (GridCollector) which is currently
able to fetch files across the grid and deliver a selective amount of
event to the application.
- For monitoring of the
resource consumption on the fabric, the focus was on a predicted stable
and community accepted solution: Ganglia monitoring tools at each of our
two major sites, coupled with our work on the MDS publishing of the
Ganglia information, now provide the first corner stone for resource
monitoring on the fabric.
- Further monitoring
tools such as the MonaLisa[6]
framework was deployed at both BNL and NERSC and new version are being
periodically tested as they appear (and enhanced requested feature are
included)
In addition, several
activities were started in as collaboration between STAR and ITD as well as
with remote Computer Scientists and CS research groups such as the SDM[7]
group at LBNL (reliable data transfer using SRM[8],
Replica Registration Service or RRS to name only those) . For the coming year,
we in fact intend to
- help the development, test and use replica
registration service components (RRS),
- freeze of our fabric monitoring approach and/or
further enhance its capabilities as needed
- work on setting up a multi-site production
environment using heterogeneous computing resources for
Moreover, we believe we will
be in a position to open the Grid to our users for running analysis jobs by
2004 and to this end, will need a user certificate and VO management system,
functional grid-ware database solutions, logger utilities at application level,
error recovery and job tracking system some of which are part of ongoing
investigative activities.
We feel that the
incorporation of the Grid into the daily activities of such a large experiment
will provide a path for other scientific groups at BNL to follow as the Grid
technologies permeate the scientific computing landscape.
STAR/ITD and iVDGL
Production level data replication across the US Grid has been demonstrated as being a successful first step, but data production/data mining remains at an embryonic stage (at the developer level) and requires hardening and interoperability before a generalized use amongst the user community can occur. The ongoing RHIC Physics program, with increasing data throughput, drives the need for the use of distributed resources available across an international set of collaborating institutions. In fact, no less than four institutions will join the STAR Grid efforts in the coming months and put us in a good position to test the many facets of the actual grid infrastructure that is, not only under a controlled Monte-Carlo like driven testing but also at user analysis job level and interactive analysis using the Grid.
The STAR/ITD collaboration would greatly benefit from joining the iVDGL project and we believe this benefit will be mutual.
· Several ongoing activities are overlapping with iVDGL interest (monitoring, testing and/or deployment of VDT as a common toolkit, use of MDS etc…) and bringing our experience and feedback to the source of the activities, while benefiting from iVDGL’s greater awareness of direction and development, will lead to a net benefit by avoiding the duplication of efforts.
· IVDGL has extensive experience in packaging and distribution of standard toolkits. We hope to learn from this experience in our distribution of software components to organizations both on and off BNL.
· While iVDGL is aimed toward driving the Grid to an every day production use of Petabyte scale, we believe that, as an active on going experiment, STAR can truly bring this goal to reality: we are an experiment with real PetaByte scale data samples every year, and since we are resource constrained we do need to migrate both our production and user analysis to a Grid infrastructure. We believe that existing experiments, with a large number of potential early users of the Grid, offer a superb and immediately available test-bed for Grid infrastructure and developments.
· We believe we can play an important role in projects such as Grid3. While shaped around providing infrastructure and services for LHC-oriented production, we firmly believe that early running experiments offer a unique (user) environment in which base components of the Grid can be stress tested and therefore, allow for catching possible problems at an early stage. We see this as a major strength of our possible collaborative efforts as robustness and scalability issues of the components can be immediately assessed by our community.
· Sharing the same facilities with other experiments, participation to existing collaboratory pilot projects is fundamental to us as we can cross benefit from development, experience and manpower put into other areas, allowing for a faster convergence toward our Grid activity goals.
· Furthermore, we view the participation of ITD as a crucial ingredient toward a long term successful Grid deployment. ITD has showed a strong interest in Grid activities and joined the STAR team in the PPDG effort. In order to broaden the scope of Grid activities outside the Physics department where it is currently confined, and to develop a real Brookhaven Grid, we believe a strong participation of the STAR/ITD team in iVDGL activities to be an essential step toward establishing a driving force which will propel us all toward success. It is clear in our mind that since the Grid activities are pushing the frontier of networking, security, software application in general, it is natural to include in the long term planning our local IT team having both most experience in dealing with a large heterogeneous community. Participation in issues such as security is essential to ensure compatibility between the Grid ideals and the DOE cyber-security laboratory regulations. As a facility support group, participation and teaming with iVGDL / iGOC is natural.
Conclusion
The STAR and ITD teams have already begun to work on a production grid environment at BNL. At this point in our development, we feel it appropriate to seek out and join collaborations such as iVDGL for the mutual benefit of all. We look forward to working with iVDGL in the near future.