Archival systems

The High Performance Storage System (HPSS) provides data archive services for the HPC system Yellowstone, as well as for divisional servers at other NCAR laboratories and UCAR programs. HPSS usage grew throughout FY2015 with over 12 PB of new data and 32 million files added to the archive during the year.

HPSS growth
HPSS growth in petabytes.

CISL’s HPSS has supported offsite duplication for CISL’s Research Data Archive (RDA) based on CISL’s FY2013 business continuity plan. The primary data copy is resident at the production NWSC facility, while the secondary copy is stored at the Mesa Lab Computing Facility (MLCF) in Boulder. This ensures both data replication and geographic separation for disaster recovery support. In FY2014, this service was also extended to EOL’s high-value, irreplaceable data collections. It is anticipated that more labs and programs will make use of this service in the future.

In FY2015, the HPSS system was augmented with a further 20 T10K-D drives to augment the 30 additional drives that were installed in FY2014.

A limited migration from Oracle T10K-C tapes to T10K-D formatted tapes was conducted in FY2015. The D-tapes offer a 60% capacity increase over the C-tapes. Further migration will take place in FY2016. All Oracle T10K-B equipment residing at the MLCF was decommissioned.

After continued analysis of the growth and performance of the archive and assessment of the technology landscape, and also to benefit from early experience with the NWSC-2 supercomputer, CISL postponed the release of the tape library procurement until mid-2017. To meet the expected data load increase from the new supercomputer in 2017, further analysis and projection exercises were conducted to size an augmentation of the current HPSS equipment.

To help manage cost, CISL implemented and currently executes policies for orphaned file processing to ensure that older data either leaves the system or is adopted by funded projects to cover the cost of keeping them. In FY2015, a further step was taken to target abandoned files for deletion, prior to the next tape technology migration.

The HPSS effort supports CISL’s computing imperative for hardware cyberinfrastructure by deploying a production HPSS instance that supports NWSC archival services requirements. The NCAR HPSS is managed by CISL under the UCAR/NSF Cooperative Agreement and is supported by NSF Core funds and CSL funding.