File replication
|
|
File replication |
STAR's primary need is for large-scale (~100 TB in 2003) replication of production-level data from BNL to LBNL. Working with the Scientific Data Management Research Group (SDM) STAR has adopted their Storage Resource Management (SRM) tools for this purpose. These tools enable replication from tertiary storage at BNL to a local disk cache, across the wan via gridFTP to a local disk cache at LBNL, and into tertiary storage at LBNL. This three step transfer looks like a simple one step copy as the SRM tools take care of queuing of multi-file requests, scheduling of file transfer tasks from/to HPSS as well as concurrent gridFTP transfers, disk cache management, and recovery from temporary failures of these transfer services. In May 2002 sufficient bandwidth (2-3 MB/s) and gridFTP reliablility had been achieved to use these tools for the majority of STAR data replication needs. Future work in this area will focus on improving the bandwidth, implementing replica catalogs and possibly utilizing the Grid Data Mirroring Package (GDMP).
Eric Hjort - page was last modified