Enhance data analysis and visualization software

CISL develops and supports two data analysis and visualization software packages: The Visualization and Analysis Platform for Ocean, Atmosphere, and Solar Researchers (VAPOR) and the NCAR Command Language (NCL).

The VAPOR project is an open source software development effort aimed at improving the ability of researchers in the Earth System sciences to interactively analyze and interpret results arising from numerical modeling. VAPOR’s unique features include its use of a wavelet-based, progressive-access data model that permits exploration of some of the largest simulation outputs using only desktop computing resources; a feature set and user interface that are focused on the needs of the Earth System sciences community; and a strong emphasis on supporting both qualitative and quantitative data analysis. The VAPOR package has a community of over 9,000 registered users worldwide, and to date has been cited over 280 times.

NCL visualization
This NCL graphic shows two regional model domains with different grid resolutions mapped onto one globe. It was written by Karin-Meier Fleischer (DKRZ) for the NCL User Guide, which has an extensive tutorial on NCL graphics and a smaller section on PyNGL and PyNIO. This graphic shows a variety of NCL’s capabilities.

NCL is an open source data analysis and visualization environment developed in close collaboration between CISL, NCAR climate modeling and weather research groups, and the university and broader geoscience communities. PyNIO and PyNGL are Python modules based on NCL’s file input/output and visualization capabilities. WRF-Python provides a comprehensive Python interface to the WRF-ARW computational and visualization capabilities in NCL. These science-driven, well-supported, free tools enable scientists to effectively read, analyze, and visualize a wide variety of complex geoscientific data formats on a variety of computing platforms. NCL has been embraced broadly across the international geoscience community spanning research, education, operational, military, government, and commercial organizations. It is used for creating publication-quality visualizations, analyzing climate model data, and real-time data display at operational centers. The Python tools are aimed at exposing NCL’s unique capabilities to a mainstream language that has widespread adoption in the scientific community.

A CISL strategic goal is expanding the productivity of researchers in the atmospheric, geospace, and related sciences through advanced computing and data services. VAPOR’s Big Data capabilities make it unique among advanced visualization tools and satisfy a critical need for computational geoscience, particularly in the areas of weather prediction and climate and ocean modeling. NCL and related Python tools address disparate and evolving research issues in the climate, atmospheric, and oceanographic community. For example, the effects of heat waves, droughts, and evapotranspiration on humans and agriculture. Scientists can produce results more quickly and effectively because these tools facilitate decision making, improve understanding, and stimulate insights from data sets that are being produced in varieties and volumes far exceeding anything they’ve ever had to manage before.

The VAPOR team’s FY2017 efforts were focused primarily on preparing the release of the third major version of the package, VAPOR3. This involved a complete refactoring of the VAPOR code base. VAPOR3 provides a far more general data model, capable of supporting both structured and – for the first time – unstructured grids. When VAPOR3 is released in early FY2018, it will support both MPAS and ICON model outputs sampled on their native, unstructured meshes. In addition to a much-improved data model the new package will also offer a more modular architecture, designed to facilitate extensibility by third parties, and a new and much simplified graphical user interface (GUI). In parallel with VAPOR3 development, Version 2.6 of VAPOR2 was released in the spring. Version 2.6 includes improved integration with Python, particularly with the Matplotlib library, and a new variable statistics module.

The NCL team released major versions of NCL, wrf-python, PyNGL, and PyNIO in FY2017. NCL version 6.4.0, released in February 2017, includes over 100 new functions for crop, heat stress, extreme value statistics, bootstrapping, and evapotranspiration; significant improvements to HDF5, NetCDF4, and GRIB readers; an overhaul of graphical options to allow for RGB/A color (transparency); and a new 260+ page NCL User Guide that was written in collaboration with a visualization team at DKRZ. The first official release of wrf-python was in March 2017, and it provides more than 30 diagnostic calculations, several interpolation routines, and utilities to help with plotting via cartopy, basemap, and PyNGL. PyNIO 1.5.0 and PyNGL 1.5.0 were released in September 2017. PyNIO includes improved support for reading HDF5 compound and group data, while PyNGL was overhauled to support the same 32-bit graphical display that’s in NCL. Other significant efforts started in FY2017 include the port of PyNIO and PyNGL to Python 3.5 and a major refresh to NCL’s map databases via shapefiles. All of these software packages are now being released under conda-forge, a community-driven packaging for conda with a high focus on scientific packages. In the period October 2016 to September 2017, NCL was downloaded 28,960 times with a monthly average of 45,346 unique visitors to its website. Wrf-python was downloaded 23,164 times with a monthly average of 665 unique visitors to its website. PyNIO was downloaded 14,128 times and PyNGL 4,483 times, with a monthly average of 2,883 unique visitors to their website.

The VAPOR project is supported by NSF Core funds, a subaward from the University of California at San Diego, NSF 54067252, and a grant from the Korean Institute for Science and Technology Information. The NCL and related Python tools project is supported by NSF Core funds.