Imperative IV
Provide Comprehensive Data Services, Open Access, and Long-Term Stewardship of Data
One of NSF’s core expectations in the NSF - UCAR Cooperative Agreement focuses on data issues, specifically calling for NCAR to “serve as stewards of high quality scientific data on behalf of the community through maintenance, enhancement and curation.” For EOL, this charge falls on the Computing, Data and Software Facility (CDS). CDS is responsible for developing and maintaining EOL’s computing infrastructure, data and metadata services, collaborative tools, and software engineering, all of which are integral to Imperative IV. CDS serves as the umbrella for all data management activities in EOL, and takes a proactive approach in working with PIs to meet NSF’s requirement that, beginning in January 2011, grant proposals include comprehensive data management plans.
Field Catalog 2.0
Development work continues on the Field Catalog 2.0, a major upgrade of the software that has served the lab and the scientific research community for the last 17 years. During FY 2012, major components of the new software were developed, tested and used to support field campaigns, including Catalog Maps (described below), product ingest and synchronization, and monitoring components. These new developments supported the DYNAMO, TORRERO and DC3 field campaigns. New ingest and synchronization software allowed us to reduce the latency in field catalog products availability to less than 30 seconds from reception. It also included for the first time an ability to ingest products according to a defined priority system, ensuring that vital products needed for real-time decision-making were available without delay. For DYNAMO, the synchronization component allowed CDS to set up two-way mirroring between a local catalog set up in the Maldives and one running in Boulder, and using a user-defined set of products that were based on priorities halfway around the world. This system provided fast access with minimal delays to catalog products for users at the field site while ensuring robust collection of all project related products. Also in FY 2012, we worked to develop monitoring software with the assistance of an EOL Summer Undergraduate Program for Engineering Research (SUPER) intern. This software will allow CDS to automatically watch all field catalog product feeds and related system services, and to immediately detect problems and notify support staff. This monitoring component has been demonstrated and will be completed and implemented prior to the next supported field campaign.
Catalog Maps
The Catalog Maps tool, a component of the field catalog, was developed during 2012 to provide real-time access to geographic information system (GIS) information and overlay capabilities for the broad spectrum of project participants, particularly those using mobile devices or using cellular networks to access the internet. This tool was rapidly developed ahead of the Deep Convective Clouds and Chemistry (DC3) campaign and was first used in the field for that project. Based on OpenLayers technology and taking full advantage of Web Map Service (WMS) protocols, Catalog Maps is designed to work across all major computer operating systems and to support a large number of users efficiently. It provides access to major operational and research products from a campaign, including satellite, radar and lightning data as well as aircraft and research instrument locations, tracks and data plots. Development work continues on Catalog Maps to incorporate replay capabilities and to refine the interface for touch screen devices.
Figure 29: The Mission Coordinator (MC) display during DC3, showing radar based location of storms, satellite overlay of clouds, the GV flight track in green, and the NASA DC8 flight track in red. Items that can be displayed include NEXRAD radar, CAPPI at flight level, research radards, satellite (IR/vis), lightning, (NAPLN/LMS), camera from the GV, flight track with winds, flight plan, and VORs and other navigational points.
Lidar Radar Open Software Environment (LROSE)
Atmospheric researchers make extensive use of scanning and profiling remote sensors, including microwave radars, wind profilers, lidars and sodars. However, extracting the full value from measurements made with these sophisticated instruments depends on having good software tools, and on shared analysis among the community of users. NCAR has a long history of such software and community format development for scanning radars. Capitalizing on this history, and appropriate for a National Center, LROSE will create a core software and open-source software exchange. This exchange will be facilitated by NCAR, but software development and maintenance will occur throughout the user community. This exchange would function well beyond the scanning radar community to benefit users of wind profilers, some lidars, and possibly other remote sensors. Such a shared approach could be used for all software used by NSF’s LAOF community.
Developed in FY 2012, the LROSE proposal would build on 25 years of software and standards created by NCAR and UNIDATA. This software has been a foundation for the weather radar research community, but it has become clear in recent years that NCAR cannot expand or even continue to maintain the existing suite of software in the rapidly changing technological environment. LROSE’s NCAR-Community hybrid calls for NCAR to develop data formats and core software, and then enables and encourages the user community to participate by developing application modules and algorithms. These modules would extend the core software to meet developer needs for specialized data quality control, scientific analysis, display, and data integration, and ideally would be useful to others in the community. This hybrid approach is analogous to the successful approach used by the Weather Research and Forecasting model (WRF) in which NCAR designs and develops data formats and core software, while the user community develops modules written in efficient high-level languages, such as MatLab® , IDL®, NCAR Command Language (NCL), and Python.
The LROSE approach would extend beyond scanning weather radar research software to also facilitate wind profiler and other remote sensor research. For instance, converters for common commercial wind profiler formats could greatly broaden the use of non-LOAF wind profiler datasets. The LROSE hybrid model will allow flexibility in addressing, and timely response to, changing needs in today’s world of quickly evolving software and instrument development.
Data Citations and Digital Object Identifiers (DOIs)
Federal agencies, professional societies, and research organizations in the geosciences are moving towards requiring researchers to formally cite data (or digital resources) that lead to a given research result. However, before digital resources can be cited, they must be designated as citable objects with unique identities. The most common type of unique identifier used within current global scholarly communication systems is Digital Object Identifiers (DOIs). DOIs provide unique identifiers/locators for web-based objects (they are now most commonly assigned to journal articles), and are an integral component of data citations. They are designed to overcome the inherent unreliability of URLs by providing persistent locators for internet-based resources.
A NCAR/UCP Data Citation Group consisting of representatives across the institution was formed in the Summer/Autumn of 2010. EOL is well represented in this Group, and in FY 2012 begain working towards a policy to handle EOL datasets. NCAR has also established membership with EZID, a California Digital Library service, in order to enable NCAR/UCP groups to assign DOIs to data sets. In FY 2012, EOL began testing EZID within the EOL Metadata Database and Cyberinfrastructure (EMDAC), including enhancing our metadata database, and we are developing plans for pilot implementations. This work will continue in FY 2013.
NCAR Membership in the Federation of Earth Science Information Partners (ESIP)
Under EOL leadership, NCAR became a member of the Federation of Earth Science Information Partners in FY 2012. From the ESIP web site:
“the Federation of Earth Science Information Partners (ESIP) is an open networked community that brings together science, data and information technology practitioners... In this forum, practitioners work together on interoperability efforts across Earth and environmental science allowing self-governed and directed groups to emerge around common issues, ebbing and flowing as the need for them arises. These efforts catalyze connections across organizations, people, systems and data allowing for improved interoperability in distributed systems. By virtue of working in the larger community, ESIP members experience the network effect, which enables more coordinated cyberinfrastructure across domain-specific communities. Using this open, community-based, discipline and agency neutral approach, the ESIP Federation has a 14-year track record of success and continued growth.”
EOL sponsored three staff members at the summer 2012 ESIP meeting, where there were productive exchanges and discussions of future collaborations related to data services and cyberinfrastructure for observing systems.
EarthCube
The goal of the NSF-sponsored EarthCube initiative is to transform the conduct of research by supporting the development of community-guided cyberinfrastructure to integrate data and information for knowledge management across the Geosciences. EOL staff participated in EarthCube Charrettes One and Two in FY 2012 and helped define a community framework, which includes the domain of lower atmospheric observing systems, for the initiative. EOL also submitted two vignettes, or case studies, to the EarthCube Workflow Subgroup: “NCAR/Earth Observing Laboratory Field Project Data Services Lifecycle Workflow” and “A Real-Time Operations Paradigm: Real-Time Science Mission Situational Awareness Workflow.” Workflow methodologies have grown in importance as a result of the greatly expanding complexity and scale of large computing activities, which touch on data analysis and modeling, often performed throughout cloud computing infrastructure. EOL provides extensive and sizeable observational datasets and so we are commonly at the front end of a workflow. It is therefore crucial for EOL to effectively integrate with these new computing paradigms.
Advances in Observing System Software
In FY 2012, EOL continued to innovate in observing system software on many fronts, a few of which are discussed below.
- The Mission Coordinator (MC) has evolved into a “must-have” application for all NSF/NCAR research aircraft programs. The MC, which began as a SUPER intern project (see Imperative IV), has evolved into a highly extensible and user-friendly tool for situational awareness and mission-planning. In FY 2012, the DC3 project benefited from the MC’s new and greatly extended menu of customized real-time products delivered to flight crews and ground based scientists.
- EOL was proud to share the Software Defined Digital Down-Converter (SD3C) with the Center for Severe Weather Research (CSWR) in FY 2012. SD3C was developed at EOL as a radar processor for EOL radars and wind profilers, and we are now working with CSWR to employ it as a signal processor for the Doppler on Wheels.
- Remote instrument control, especially for airborne platforms, has long been a goal for EOL. This became a reality in FY 2012 with the development of remote operations for the AVAPS dropsonde system, deployed on the NASA Global Hawk. This work is being expanded and adaptated for use with the NSF/NCAR GV aircraft (see Imperative III). Development of a general-purpose communication architecture began in FY 2012; this will allow PIs to remotely monitor and control their own instruments on the GV. AVAPS will be the first measurement platform to utilize this system.
- The University of Wisconsin-developed High Spectral Resolution Lidar (HSRL) was integrated on the GV for airborne observations in FY 2012. However, the software delivered with the instrument was prototypical in nature, and so in FY 2012 EOL collaborated with the U. Wisconsin developers to successfully refactor and migrate the software into a more robust and modern form.
- The first deployment of the Compact Atmospheric Multi-species Spectrometer for HIAPER (CAMS) occurred during DC3 and used software built on an Object Oriented (OO) LabVIEW framework. EOL has become a leader in exploring and developing the OO LabVIEW methodology, which brings improved software architecture and practices to the LabVIEW ecosystem.