Provide networking services

Networking infrastructure is essential technology vital to UCAR’s ability to function and prosper in a rapidly evolving scientific and technical environment. Networking infrastructure enables many aspects of the scientific enterprise to flourish: business processes, scientific investigations and analysis, communication, global collaborations, and educational and outreach missions. Network infrastructure is literally the backbone of all other IT infrastructure and services. A sound and reliable network infrastructure is critical to building stable IT infrastructure at the higher levels.

Networking is a critical component of cyberinfrastructure. Networks are interconnected and interoperate at the campus, metropolitan, regional, national, and international levels. Networking is a global endeavor. Being well connected has become a requirement for successful business operations, but it is especially important for a national research center such as UCAR/NCAR.

CISL plans, engineers, installs, operates, maintains, develops strategy, and performs research for fast, reliable, and flexible networks that support data services at the campus, metropolitan, regional, national, and international levels. NETS provides a vital service to NCAR’s research communities by linking scientists to supercomputing resources and each other. These activities are essential for the effective use of UCAR/NCAR scientific resources, and they foster the overall advancement of scientific inquiry.

At its core, CISL is a laboratory dedicated to providing advanced cyberinfrastructure to propel world-class science. CISL services evolve in response to changes in the underlying cyberinfrastructure technologies and the scientific demands of the community.

NETS pursued these Local Area Network (LAN) and general support projects in FY2016:

  • UCAR network infrastructure re-cabling
  • WASP inventory system
  • Geographic Information System (GIS)
  • Softphones
  • Voice over Wifi
  • Cellular support
  • ML Room 034 remodel and NETS shop relocation
  • Network monitoring
  • Netflow
  • Extraview
  • Filemaker
  • Business continuity
  • Everbridge Emergency Notification System (ENS)
  • UPS
  • Grounding
  • Wireless networking
  • Voice Over IP (VoIP)
  • Collocation Facilities Management (CFM)
  • CO2 Monitor networking
  • EOL field project wireless testbed
  • HAO High Altitude Chamber
  • ML Climate Exhibits
  • ARUBA Clearpass design
  • MLDC (ML-29) infrastructure design
  • Cisco VG320 voice gateway upgrade
  • IPv6 UCAR deployment
  • Spring and fall power downs
  • Equipment maintenance
  • Budget support
  • Purchasing support
  • HAO MLSO LAN upgrade design and implementation
  • OpenDCIM
  • Patchmanager
  • NWSC Module A & B design
  • NWSC2 procurement
  • Maintenance of NETS servers
  • Replaced Smartsheets with Google Docs
  • Migrate to CISL domain
  • CISL and NCAR Director’s Office Chromebox deployments
  • Unidata move inside security perimeter

NETS pursued these Metropolitan Area Network (MAN) projects in FY2016:

  • Boulder Point-Of-Presence (BPOP)
  • Boulder Research and Administration Network (BRAN)
  • BRAN Vassar Fiber Cut     
  • SHEPC inter-building cabling and agreement
  • BRAN Table Mesa Flood Mitigation

NETS pursued these Wide Area Network (WAN) projects in FY2016:

  • Front Range GigaPoP (FRGP) ongoing management and engineering
  • Bi-State Optical Network (BiSON) ongoing management and engineering
  • DREAM 20G upgrade
  • 100G NWSC/FRGP path
  • New FRGP participants:
    • The State Higher Education Policy Center (SHEPC), incorporated under Colorado law by three nonprofit corporations: National Center for Higher Education Management Systems (NCHEMS), State Higher Education Executive Officers Association (SHEEO), and the Western Interstate Commission for Higher Education (WICHE)
    • Jefferson County School District
    • Johnson and Wales University
  • Internet2 Gender Diversity Initiative Co-chair
  • Internet2 Network/Connector Liaison
  • Western Regional Network (WRN)
  • WRN NOAA support
  • NOAA Research Network (NWAVE)
  • NOAA FRGP TIC design and implementation
  • 910 colocation renewal
  • Level3 Fiber Lease Renewal
  • Colorado Department of Transportation (CDOT) Higher Education Fiber Agreement
  • The Quilt Project – National Regional Networks Consortium
    • Jeff Custard – Executive Committee/Chair until 2/19/16
    • Marla Meehl – Nominations Committee
    • Marla Meehl – Finance Committee
    • Marla Meehl – CC*IIE Regional Collaboration Working Group
  • NSF CC*DNI UH Mauna Loa Instrument Proposal to provide high speed wireless networking support
  • CSU-P NSF CC* 10G research lambda
  • Added two CSU-FC 10G lambdas
  • BPoP TCOM Upgrade
  • ADVA ALM Testbed
  • Submitted NSF CICI Rocky Mountain Regional Internet Security Collaboration (RMRISC) Proposal - not funded

NETS pursued these special projects in FY2016:

  • NSF RMCMOA CC*IIE Grant and WINS Supplement
    • Two RMCMOA workshops
      • LBNL CIRA Report on Engaging Researchers
    • Women in IT Networking at Supercomputing (WINS) awarded five women to attend and participate in SCinet and SC 15
    • No Cost Extension to 2/28/17
    • Change of Scope
  • NSF WINS Three-year Proposal - unsolicited
    • Evaluated 33 submissions
    • Awarded eight women to attend and participate in SCinet and SC 15
  • Westnet meeting support
    • January 2016
    • June 2016

NETS will continue to provide support and enhancements for all of these essential networking services.

Detailed project descriptions appear below for three of these projects: NWSC data center network design and implementation, Mauna Loa Network collaboration known as MLONET, Fiber expansion (Level3 fiber lease renewal, CDOT fiber lease agreement, SHEPC fiber build).

NWSC data center network design and implementation

This is an ambitious phased project to install a Juniper QFX10008 (Mule) capable of 40 and 100 Gbps and upgrade all Juniper 3500s to much more capable Juniper QFX5100s.

This project was planned in 2015 and was completed in fall 2016. This is a full refresh of the data center network to support the new supercomputer, Cheyenne, and the new GLADE storage and analysis system.

The NETS team replaced eight QFX3500 switches with QFX5100s, added an additional five QFX5100s to meet the requirements of the project, and added 240 10 Gbps ports. In its current configuration, Mule can provide up to 100 40 Gbps ports and two 100 Gbps ports, scaling up to 96 Tbps of throughput. NETS worked closely with DASG to procure, design, and install the QFX10008 Mule to provide 40 Gbps Ethernet connections between 20 “old” GLADE-1 and 32 “new” GLADE-2 hosts.

Using a phased approach, the NETS team created two virtual chassis and also replaced the Cisco 1 Gbps switches in the data center with Juniper EX4300s. Most of the computing devices in the data center require redundant connectivity. Using virtual switches in the design allows these devices to connect to two or more switches, which provides a more resilient network, mitigates outages caused by any individual device, and allows component upgrades with less downtime. During the project phases, NETS, CASG, and ISGC teams moved over 500 ports from old equipment to the new network.

NWSC network update
This diagram shows the high-level design for the NWSC data center network upgrade.

Mauna Loa Network collaboration known as MLONET

MLONET is a joint project between the University of Hawaii (UH), NCAR/UCAR, and NOAA to upgrade the WAN for instruments housed on Mauna Loa, Hawaii, including the HAO Mauna Loa Solar Observatory (MLSO). This project is funded in part by NSF award ACI-1541471 “CC*DNI Instrument: High Performance Reliable Network Access to Mauna Loa Observatory Science Instruments” in the amount of $398,207 awarded to UH with support from NCAR/UCAR and NOAA. Mauna Loa has suffered with costly and inadequate networking for many years, and the difficult-to-reach location limits the networking options and increases the cost. The MLONET team evaluated options, reviewed the high-speed wireless design, and managed the installation of the system that has gone active. In addition, the NETS team took advantage of a facility upgrade project at MLSO to design and install a new Juniper router/switch, new LAN cabling and wallplates, a new wireless LAN system, and new fiber optic networking to tie into the new WAN wireless system. This upgrade moves the MLSO network to NETS standards and increases WAN bandwidth from 100 Mbps to 400 Mbps. This has been an excellent collaboration between UH, NOAA, NETS, and HAO.

Mauna Loa network
Images of LAN and WAN networking equipment and infrastructure used in the Mauna Loa Network collaboration.

Fiber expansion (Level3 Fiber Lease renewal, CDOT fiber lease agreement, SHEPC fiber build)

Center Green fiber expansion
Fiber pathway connecting SHEPC to the UCAR Center Green 2 fiber.

Extensive expansion and renewal of WAN fiber optic cabling has taken place this year. Owned/leased fiber optics are the preferred method of constructing LAN, MAN, and WAN networks. NETS has been able to greatly expand the amount of fiber owned or on long-term lease for the use of NCAR/UCAR, the FRGP, and BiSON. NETS completed a very long negotiation with Level3 to renew valuable fiber leases with no cost increase. These paths provide critical physical network segments on the BiSON and UCAR networks. NETS has worked for a number of years to establish a relationship with the Colorado Department of Transportation (CDOT), which controls extensive fiber optic cabling in the state and federal right of ways. The partnership has been solidified after the signing of a 10-year agreement for multiple fiber paths including Boulder to Golden on US 93, Boulder to Denver on US 36, Denver to Stateline on I-25, and select paths in Pueblo for the use of CSU-Pueblo. NETS also succeeded, after a long negotiation, in partnering with the City of Boulder for a right of way between the SHEPC facility on Center Green and CG2. This allowed SHEPC to join the FRGP.


NETS activities are primarily supported through UCAR Communications Pool indirect funds, the FRGP/BiSON, NSF Core funds, and NSF awards ACI 1440642 and ACI 1640987.