Support strategic use of CISL resources

The CISL production supercomputing environment supports special computational campaigns for ongoing and short-term computational projects, all via a priority-based and near-real-time job scheduling mechanism. These campaigns are managed to minimize the impact on the production computing delivered to NCAR, university, and CSL scientists.

Wind flow visualization
Wind power, while clean and renewable, requires constant load balancing due to variable weather conditions. These conditions must be met with state-of-the-art forecasting methods to maximize load-balancing efficiency. As part of their ASD project on Cheyenne, Pedro Jimenez Muñoz and Branko Kosovic of NCAR’s Research Applications Laboratory developed new techniques in numerical weather prediction to better optimize load balancing. Visualizations of their work by CISL’s Scott Pearse show how winds behave on the leeward side of Mt. Hood. These winds generate a wake that affects Oregon wind farms, causing swings in power that require dynamic load balancing.

In FY2017, the special campaigns included 10 Accelerated Scientific Discovery (ASD) projects on the Cheyenne system. The ASD initiative provides dedicated, large-scale computational resources to a small number of projects for a very short time period, usually two or three months following acceptance of a new HPC system. These projects are selected to help put the new system through its paces and pursue scientific objectives that would not be possible through normal allocation opportunities. Altogether, 11 ASD requests were received in response to a call to the university and NCAR communities in 2016. NCAR requests were reviewed by a panel of NCAR scientists and approved by the NCAR Executive Committee; university requests were reviewed by the CISL HPC Allocations Panel (CHAP).

This table lists the ASD projects and other special computational campaigns that CISL supported during FY2017.

FY17 Special Campaign

Project Lead



ASD: Turbulence and magnetic reconnection in the heliosphere

M. Shay (U Delaware)



ASD: Project Metis—Seasonal forecasts with enhanced ocean and atmosphere resolution

B. Cash (George Mason U)



ASD: Estimating CESM high-resolution near-term climate predictability

B. Kirtman (U Miami)



ASD: Impacts of smoke aerosols on the transitions between closed and open cellular convections of stratocumulus clouds over Southeast Atlantic

X. Liu (U Wyoming)



ASD: Wind turbine performance and loading response to turbulent atmospheric inflow conditions

D. Mavriplis (U Wyoming)



ASD: Climate impacts based on a multiple-member ensemble geoengineering study with WACCM

S. Tilmes



ASD: Sub-seasonal to seasonal prediction using high-resolution CESM

J. Tribbia



ASD: Predicting near-term changes in the likelihood of climate extremes

S. Yeager



ASD: CESM2 regional climate community simulations

A. Gettelman



ASD: Emergence of the magnetic field from the convection zone into the corona

M. Rempel



ASD: Assessment of a 3D planetary boundary layer parameterization for the WRF model

P. Jimenez Muñoz



Aerosols in shallow tropical convection

A. Nugent



NMME Phase II Seasonal System

J. Tribbia



Support of IPCC Special Report on 1.5-degree C target

J.-F. Lamarque



Continuance of our real-time, high-resolution ensemble forecast effort

G. Romine



ORACLES campaign

M. Barth



Advanced study of wind gusts in hurricanes using large-eddy simulation

G. Bryan



Research and development of the National Water Model using the community WRF-Hydro modeling system

D. Gochis



From these projects, Pedro Jimenez Muñoz and Branko Kosovic from NCAR’s Research Applications Laboratory presented early results from their ASD effort at the 2017 WRF Users’ Workshop[1] and an animation by CISL’s Scott Pearse was part of the Visualization Showcase at the PEARC17 conference[2]. Their project’s goal was to help improve the high-resolution prediction capabilities of Advanced Research WRF (WRF-ARW) for wind over complex terrain. They looked specifically at Oregon’s Columbia River Gorge and how winds behaved on the leeward side of Mt. Hood (see visualization above). Their simulations show that the winds generate a wake that affects Oregon wind farms and causes power fluctuations that require them to perform dynamic load balancing.

Special campaigns
Cheyenne and Yellowstone core hours dedicated to Accelerated Scientific Discovery (ASD) and other special computational campaigns during FY2017. CISL works to accelerate scientific discovery through numerical simulation by providing a portion of its HPC systems to special campaigns. The four special campaigns not shown collectively used 320,000 additional core-hours on Yellowstone.

A number of ASD projects explored aspects of climate prediction, including Project Metis, led by Ben Cash of George Mason University. Project Metis continued a collaboration between the Center for Ocean-Land-Atmosphere Studies (COLA) and the European Centre for Medium-Range Weather Forecasts (ECMWF) in support of both centers’ ongoing efforts to understand and quantify predictability in the climate system from daily to interannual time scales. They tested very-high-resolution model configurations, with grid spacings of 9 km in the atmosphere and 25 km in the ocean, representing a major step forward with respect to current seasonal and subseasonal forecasting systems. James Kinter, director of COLA, presented early results from Project Metis at the 2017 International Computing in the Atmospheric Sciences (iCAS 2017) symposium. Of particular interest to climate prediction efforts more broadly, Project Metis has shown improved prediction skill in some aspects of climate, but successful climate prediction with higher-resolution models may demand larger ensembles – that is, even more computing resources – simply because both signal and noise increase with resolution.

These special computing campaigns serve CISL’s computing imperative to provide on-demand and real-time services support for hardware cyberinfrastructure. This work is made possible through NSF Core funds, including CSL funding.

[1] Kosovic, B., and P.A. Jimenez, 2017: Evaluation of a three-dimensional PBL parameterization for simulations of flow over complex terrain. WRF Users’ Workshop, June 12–16, 2017. Boulder, Colo.

[2] Pearse, S., P. Jimenez Munoz, and B. Kosovic, 2017: Visualization of NCAR's Wind Forecast Improvement Project 2. Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact (PEARC17). ACM, New York, NY, Article 77, 3 pages. DOI: