Provide networking services

Networking infrastructure is essential technology vital to UCAR’s ability to function and prosper in a rapidly evolving scientific and technical environment. Networking infrastructure enables many aspects of the scientific enterprise to flourish: business processes, scientific investigations and analysis, communication, global collaborations, and educational and outreach missions. Network infrastructure is literally the backbone of all other IT infrastructure and services. A sound and reliable network infrastructure is critical to building stable IT infrastructure at the higher levels.

Networking is a critical component of cyberinfrastructure. Networks are interconnected and interoperate at the campus, metropolitan, regional, national, and international levels. Networking is a global endeavor. Being well connected has become a requirement for successful business operations, but it is especially important for a national research center such as UCAR/NCAR.

CISL plans, engineers, installs, operates, maintains, develops strategy, and performs research for fast, reliable, and flexible networks that support data services at the campus, metropolitan, regional, national, and international levels. NETS provides a vital service to NCAR’s research communities by linking scientists to supercomputing resources and each other. These activities are essential for the effective use of UCAR/NCAR scientific resources, and they foster the overall advancement of scientific inquiry.

At its core, CISL is a laboratory dedicated to providing advanced cyberinfrastructure to propel world-class science. CISL services evolve in response to changes in the underlying cyberinfrastructure technologies and the scientific demands of the community.

NETS pursued these Local Area Network (LAN) and general support projects in FY2017:

  • UCAR network infrastructure re-cabling
  • WASP inventory system
  • Geographic Information System (GIS)
  • Voice Over IP (VoIP) – roadmap
  • Softphones
  • Voice over Wifi
  • Cellular support
  • Network monitoring
  • Netflow – Netvizura
  • PerfSONAR
  • Infoblox
  • Extraview
  • Filemaker
  • Google Docs/Sites
  • Slack
  • Business continuity
  • Everbridge Emergency Notification System (ENS)
  • UPS
  • Grounding
  • UNIDATA GOES-R/16 ML installation
  • CISL and NCAR Director’s Office conference room VTC installations
  • Wireless networking
    • ARUBA Clearpass design
    • Visitor Network design
  • Collocation Facilities Management (CFM)
  • MLDC (ML-29) infrastructure design and preparation
  • IPv6 UCAR deployment
  • Spring and fall power downs
  • Equipment maintenance
  • Budget support
  • Purchasing support – UProcure adoption
  • Travel support – Concur adoption
  • HAO MLSO Wireless WAN activation
  • OpenDCIM
  • Patchmanager
  • Maintenance of NETS servers
  • Migrate to CISL domain
  • ML Climate displays
  • Mesa Lab Anemometer replacement – in progress
  • EOL wireless testbed
  • HAO Mauna Loa LAN upgrade
  • ML and FL core router design
  • Palo Alto firewall deployment with Security Engineering Group

NETS pursued these Metropolitan Area Network (MAN) projects in FY2017:

  • Boulder Point-Of-Presence (BPOP)
  • Boulder Research and Administration Network (BRAN)
  • Upgraded NWSC WAN legs to 100Gbps

NETS pursued these Wide Area Network (WAN) projects in FY2017:

  • Front Range GigaPoP (FRGP) ongoing management and engineering
    • Upgraded DREAM ring to 100Gbps
    • Upgraded Level3 commodity Internet connection to 100Gbps
  • Bi-State Optical Network (BiSON) ongoing management and engineering
  • Boulder CloudConnect design
  • SH 93 network design and procurement
  • New FRGP participants:
    • Education Networks of America (ENA)
    • U.S. Air Force – in progress
    • Adams12 School District – in progress
  • Internet2 Gender Diversity Initiative Co-chair
  • Internet2 Network/Connector Liaison
  • Internet2 External Advisory Group on Cyberinfrastructure Deployment, Support, and Protection
  • Western Regional Network (WRN)
  • WRN NOAA support
  • NOAA Research Network (NWAVE) support
  • NOAA FRGP TIC design and implementation – Phase 2
  • Jefferson County Public Safety Fiber Optic Network (J-FON) collaboration
  • Western State University using Western Area Power Authority (WAPA) fiber assets
  • Colorado Department of Transportation (CDOT) Higher Education Fiber Agreement implementation
  • The Quilt Project – National Regional Networks Consortium
    • Marla Meehl – Board of Directors
    • Marla Meehl – Nominations Committee
    • Marla Meehl – Finance Committee
    • Marla Meehl – CC*IIE Regional Collaboration Working Group
  • NSF CC*DNI UH Mauna Loa Instrument Proposal to provide high speed wireless networking support – implemented
  • CSU-P NSF CC* 10G research lambda
  • Added two CSU-FC 10G lambdas for new stadium
  • ADVA ALM Testbed
  • Submitted NSF CICI Rocky Mountain Regional Internet Security Collaboration (RMRISC) Proposal – not funded
  • University of Hawaii NSF CC*DNI Instrument Grant for Mauna Loa
  • Pacific Research Platform
    • Host FIONA node

NETS pursued these special projects in FY2017:

  • NSF RMCMOA CC*IIE Grant and WINS Supplement – completed 2/28/17
  • Women in IT Networking at Supercomputing (WINS) awarded five women to attend and participate in SCinet at SC16 and SC17
  • Westnet meeting support
    • January 2017
    • June 2017
  • Westnet Education and Research Collaboration (WERC) incorporation
  • Westnet IP address unwinding
  • UProcure and CONCUR adoption

NETS will continue to provide support and enhancements for all of these essential networking services. Three of these projects are described in detail below.

The year of 100 Gigabit-per-second links

FY2017 was the year of 100Gbps links for NETS and the WAN networks supported for the NCAR/UCAR/FRGP/BiSON community. The NETS team successfully designed, engineered, and implemented three 100Gbps upgrade projects:

  1. The NCAR Wyoming Supercomputing Center (NWSC) is now connected with two new 100Gbps paths on the east and west BiSON ring segments. This allows NWSC users of computing and data to access and move data at much faster speeds. The Pacific Research Platform FIONA node uses one of these paths and consistently shows excellent 40Gbps performance on the wide area network.

  2. NETS worked with the WRN team to upgrade the shared Level3 commodity Internet service to 100Gbps. This path is used by the entire FRGP including NCAR/UCAR as well as our WRN partners. The additional bandwidth accommodates growing daily usage, but also provides much-needed emergency backup if other services such as Internet2, ESnet Peering, or Google Peering should not be available. The ever-growing network bandwidth requirements require not only primary path support but also resilience in the face of security attacks and service outages.

  3. In the early years of the FRGP, a metropolitan fiber ring comprised of Level3 fiber segments – collectively known as DREAM – was created to redundantly connect the FRGP router to a fledgling National LambdaRail (NLR) network at the Level3 Gateway in Denver. Thus the FRGP backbone was born in the form of 2 x 1-Gbps Etherchannel paths. In 2009, the 2 x 1-Gig network was upgraded to 2 x 10-Gbps, and SuperDREAM was created. In 2010, a third router (910-gw-1) was introduced at 910 15th Street to create the triangle topology the FRGP has today. At that time, the backbone consisted of a single 10-Gbps link on each leg of the triangle. About two years later, the Wave Division Multiplexing (WDM) boom was in full swing, and NETS leveraged BiSON to add a 10-Gbps wave to the backbone. The resulting set of four 10-Gbps links served the FRGP well for several years.

    In FY2017, we again used BiSON to inexpensively add two 10-Gbps links to the backbone for a total of six, but NETS knew this quick fix would not stave off the need for 100-Gbps bandwidth in the backbone. Through 2016, traffic growth continued its unrelenting pace. The University of Wyoming and UCAR established 100-Gbps connections to the FRGP, and NOAA landed 40-Gbps connections at two locations. In early 2017, based on our daily 95th percentile accounting, NETS observed near 50% utilization on all three backbone legs. At this and growing levels of utilization, there is little room to absorb peak demand for live events or major software releases. The design implemented this year produced TeraDREAM: three 100-Gbps backbone links that created a triangle of high-bandwidth connectivity to alleviate backbone congestion concerns in a redundant and resilient fashion.

FRGP connectivity
This diagram shows the resulting 100-Gbps TeraDREAM ring and the FRGP connectivity enabled by the advanced services.

CDOT fiber

CDOT fiber paths
A significant portion of BiSON utilizes the valuable fiber-optic paths installed by the Colorado Department of Transportation. These include I-25 North and Golden West.

For many years, the Colorado Department of Transportation (CDOT) has installed conduit and fiber optics along state and interstate highways in their very valuable rights of way. Higher education in Colorado has had a long and fruitful partnership with the Colorado Department of Transportation (CDOT) to help install, test, repair, and in-trade utilize this valuable fiber asset. We give huge credit to Pat Burns and Scott Baily at Colorado State University (CSU), who really initiated and built the CDOT relationship before the FRGP even existed. NETS builds on that trusted and positive relationship on behalf of BiSON. It has taken time to build and gain trust with UCAR and the FRGP staff, and again CSU and others in higher education have done a lot to encourage and foster that trust. After many years of negotiation, in FY2017 NETS completed a formal agreement with CDOT – on behalf of the FRGP and higher education – for six new and important fiber paths on US 36, I-25, SH 93, I-70, the CSU-Fort Collins agriculture site, and CSU-Pueblo for connectivity in Pueblo. This fiber builds on CDOT agreements already in place with CSU and the Colorado School of Mines (CSM). The new 20-year agreement is very flexible and cost effective, and it allows for the addition of fiber paths as CDOT continues to install more fiber across the state. NETS is in the process of designing, engineering, and implementing the SH 93 fiber path to expand BiSON and diversify the paths out of Boulder and Golden to improve performance and resiliency for UCAR, CSM, the City of Boulder, NEON, NOAA, and the University of Colorado.

Mauna Loa Solar Observatory (MLSO)

The Mauna Loa Solar Observatory (MLSO) is an NSF-owned facility operated by the High Altitude Observatory (HAO) at NCAR located in Boulder, Colorado. HAO conducts research and provides community support and facilities in the following areas: Atmosphere, Ionosphere, and Magnetosphere (AIM); Long-term Solar Variability (LSV); and Solar Transients and Space Weather (STSW). Instruments and research efforts HAO currently operates include three solar telescopes on Mauna Loa that provide unique observations designed to understand the Sun’s release of magnetized plasma and energy and the impacts these variations have at Earth and throughout interplanetary space. The current suite includes: (a) The Coronal Multi-channel Polarimeter (CoMP) to determine conditions of the coronal magnetic field and plasma; (b) The COSMO KCoronagraph (KCor) to observe the formation and initial acceleration of coronal mass ejections (CMEs); and (c) The Precision Solar Photometric Telescope (PSPT) to track the variability of solar irradiance from magnetic structures in the photosphere and chromosphere.

Mauna Loa site
These photos show parts of the remote and difficult-to-network Mauna Loa site with the newly installed and fully operational high-speed wireless system.

NETS collaborated with the University of Hawaii and NOAA on the NSF grant “CC*DNI Instrument: High Performance Reliable Network Access to Mauna Loa Observatory Science Instruments” to increase the network bandwidth available to science researchers at the Mauna Loa Observatory (MLO) site in Hawaii. The MLO site is owned by the National Oceanic and Atmospheric Administration (NOAA) and hosts a range of university and national research in climate, astronomy, and space weather science domains. The 400Mbps (increased from 50Mbps) wide area wireless system was successfully deployed in FY2017 and became operational for HAO scientific research network traffic. The increase in bandwidth provided by WRN for Mauna Loa is allowing HAO to provide additional real-time data products from KCor to the Space Weather Prediction Center (SWPC) in Boulder, and it furnishes the bandwidth needed by new instruments (to be deployed in FY2018) that will provide SWPC with new forecasting data products. It will also allow University of Michigan scientists and students to use the Michigan space weather forecasting models on real-time observations. In addition to the network performance increases benefitting HAO, the NOAA and University of Hawaii research projects on Mauna Loa are also benefitting from this strong and mutually beneficial ongoing partnership.


NETS activities are primarily supported through UCAR Communications Pool indirect funds, the FRGP/BiSON, NSF Core funds, and NSF awards ACI 1440642 and ACI 1640987.