Provide data analysis and visualization resources

Great Barrier Reef flow dynamics
This figure is taken from an animation of the ocean flow dynamics that occur near the Great Barrier Reef near the northeast coast of Australia. It was produced using data from a model called CT-ROMS (Coral Triangle Regional Ocean Modelling System). CT-ROMS simulated 10 years of ocean activity in the Indonesian Throughflow, a region having the most complex ocean flow patterns in the world. The flow data are being used to better understand coral reefs through a metric called Potential Connectivity, which demonstrates how well coral reefs can re-seed their neighbors with larvae while they are being carried by these complex flow patterns. Understanding how reefs can regenerate each other has never been more imperative than it is right now because the most extensive coral bleaching event in recorded history is being observed at the Great Barrier Reef.

NCAR’s Data Analysis and Visualization (DAV) environment enables scientific workflows by providing UCAR’s research community with state-of-the-art systems tailored for the specialized needs of parallel data post-processing, analysis, and visualization.

CISL provides a portfolio of advanced computing and data services specifically tailored for the atmospheric, geoscience, and related sciences communities. While computing is a foundational element of scientific research, the facilities to process, analyze, and visualize computationally generated and observational data are equally important in scientific discovery and facilitating the understanding of natural processes. CISL’s DAV environment provides both hardware and software resources for developing and enhancing data analysis and visualization capabilities. Additionally, the DAV resources and CISL’s software development efforts help scientists analyze and understand features in large and heterogeneous data sets by developing new methods and tools such as VAPOR to extract and visualize information from such data sets.

DAV hardware resources, metrics, and status

NCAR’s DAV environment consists of two multi-node systems that began production service in late 2012. Each system is designed to complement the other in meeting the diverse needs of climate and weather DAV applications. The system named Geyser is targeted primarily at traditional interactive data set manipulation, data reduction, analysis, and visualization applications, and for large, data-intensive applications requiring graphical processing units (GPUs) and/or large shared memory. The system named Caldera is targeted for use by parallel graphics/visualization applications and computationally bound applications that can be accelerated via high-performance general-purpose graphics processing units (GPGPUs). Both DAV systems share a dedicated, high-bandwidth I/O network path to NCAR’s GLobally Accessible Data Environment (GLADE). Caldera and Geyser are also used extensively for production and on-demand regridding, data subsetting, and curation of NCAR’s Research Data Archive (RDA) holdings.

Caldera is a 16-node cluster comprised of IBM dx360 M4 nodes that are identical to Yellowstone’s compute nodes, except they are augmented with two computational accelerators, or general-purpose graphics processing units (GPGPUs). Each Caldera node contains two 8-core Intel Xeon E5-2670v2 (Sandy Bridge) processors, 64 GB of memory, and two NVIDIA Tesla K20X accelerators. Each K20X accelerator is capable of 1.31 teraflops of double-precision calculations or 3.95 teraflops of single-precision calculations. Caldera's peak double-precision floating point rate is therefore more than 47 teraflops. One hundred forty Yellowstone nodes are required to produce the same peak computation rate. After decommissioning the Intel Phi accelerators in the test system named Pronghorn, CISL repurposed twelve of its dx360 M4 nodes to augment Caldera’s job scheduler queues for use by DAV jobs which do not require GPGPUs.

Geyser is a 16-node cluster comprised of IBM x3850 X5 nodes that are each equipped with a terabyte of memory, four 10-core Intel Xeon E7-4870 (Westmere) processors, and one NVIDIA Quadro K5000 graphics adapter. The K5000 accelerator is designed for high-speed graphics rendering, with a single-precision floating point rate of 2.1 teraflops.

Unlike Yellowstone, which saw an average user utilization of over 95% during FY2016 and supports long-running batch jobs, the DAV platforms are designed for interactive applications and rapid job turnaround. Therefore, their average utilization is typically low, with bursts of high utilization. During FY2016, Caldera’s average user utilization was 18.2%, while Geyser’s average user utilization was 34.4%. While these figures are relatively low, they represent an increase in usage of 27% (Caldera) and 41% (Geyser) over the average user utilization measured in previous years, and the frequency of periods of high utilization increased dramatically during FY2016. CISL continues to monitor DAV workload and usage, and the observations are helping to guide requirements for future DAV systems.

CISL's monitoring of the Caldera system has shown that very few applications use the K20X GPGPUs. Since these computational accelerators are approximately an order of magnitude costlier than more traditional graphics processing units that are tailored to visualization and image rendering, CISL is re-evaluating the need for equipping systems with such high-end GPGPUs.

Additional details of the Geyser, Caldera, and Pronghorn systems are contained in the tables in the Production supercomputing section of this report.

DAV software

CISL continued enhancement work on its Visualization and Analysis Platform for Ocean, atmosphere and solar Researchers, which is an interactive 3D visualization environment for producing animations and still frame images. VAPOR is supported on and freely available for Linux and Windows systems. A major feature release of VAPOR (2.5) occurred during FY2016, and more new features will be made publicly available in the 2.6 release of VAPOR planned for November 2016. VAPOR developments in FY2016 provided high-resolution cartographic maps, 3D stereo rendering, parallel data conversion capabilities, new schemes for contouring, contour labeling, and coloring, data importers for the MPAS and ICON models, and a host of other changes and bug fixes. Additional developments included support for coupled model analysis, a more generalized wavelet-based progressive-access data format, enhanced plotting capabilities, animation encoding, and the release of an evaluation version of VAPOR3.

NWSC-2a procurement

Because the existing DAV systems are four years old, CISL will conduct the NWSC-2a procurement during FY2017 to acquire a production-quality data analysis and visualization platform that will replace the existing Geyser and Caldera systems. NWSC-2a will focus primarily on anticipated DAV workload requirements of the NCAR user community in the 2018-2022 timeframe (a companion NWSC-2b procurement will focus on the acquisition of a computationally accelerated platform). We anticipate that the NWSC-2a system will include GPUs, large shared-memory nodes for in-memory computation, and non-volatile memory, SSD, and/or burst-buffer hardware and software for accelerating the I/O operations critical to the analysis of large data sets. As of end-FY2016, CISL expects the new NWSC-2a system to be placed into production in early 2018.


NCAR’s DAV environment and services are supported by NSF Core funds including CSL funding. VAPOR is funded by the National Science Foundation (Grants 03-25934 and 09-06379, ACI-14-40412) and by the Korea Institute of Science and Technology Information (KISTI).