Network engineering and telecommunications

Networking infrastructure is essential technology vital to UCAR’s ability to function and prosper in a rapidly evolving scientific and technical environment. Networking infrastructure enables many aspects of the scientific enterprise to flourish: business processes, scientific investigations and analysis, communication, global collaborations, and educational and outreach missions. Network infrastructure is literally the backbone of all other IT infrastructure and services. A sound and reliable network infrastructure is critical to building stable IT infrastructure at the higher levels. It is the goal of the Network Engineering and Telecommunications Section to provide this fast, robust, and flexible infrastructure to support all other IT services.

Networking is a critical component of cyberinfrastructure. Networks are interconnected and interoperate at the campus, metropolitan, regional, national, and international levels. Networking is a global endeavor. Being well connected has become a requirement for successful business operations, but it is especially important for a national research center such as UCAR/NCAR. Networking provides a vital service enabling all functions at UCAR. UCAR’s scientific mission is enabled and supported by networking. UCAR’s business operations, including interactions with funding agencies, also depend on networking. Collaborative science would not happen today without networks, just as UCAR’s business operations would not function.

The Network Engineering and Telecommunications Section (NETS) plans, engineers, installs, operates, maintains, develops strategy, and performs research for NCAR and UCAR’s state-of-the-art data networking and telecommunications facilities. NETS provides a vital service to NCAR’s research communities by linking scientists to supercomputing resources and each other. These activities are essential for the effective use of UCAR/NCAR scientific resources, and they foster the overall advancement of scientific inquiry. This work supports CISL’s computing imperative to provision hardware cyberinfrastructure for the atmospheric and related sciences. It also supports CISL’s computing frontier of center virtualization by providing infrastructure for science gateways.

NETS pursued these Local Area Network (LAN) and general support projects in FY2015:

  • UCAR network infrastructure re-cabling
  • WASP inventory system
  • GIS
  • Softphones
  • ML Room 034 remodel and NETS shop relocation
  • Cellular support
  • Network monitoring
  • Netflow
  • Extraview
  • Filemaker
  • Multicast support activities
  • Business continuity
  • Everbridge Emergency Notification System (ENS)
  • UPS, grounding, wireless networking, IPT, Collocation Facilities Management (CFM)
  • ML 29 infrastructure design
  • IPT server replacement
  • Spring and fall power downs
  • Cisco 6500 to 4500 replacement
  • Equipment maintenance
  • Budget support
  • Purchasing support
  • HAO MLSO network upgrade design
  • OpenDCIM documentation
  • NWSC Module A & B design
  • NWSC2 procurement
  • CG1 Library remodel
  • EOL Executive Conference Room
  • ARUBA Clearpass design
  • Maintenance of NETS servers
  • Replaced Vidyo systems with Chromebox
  • Replaced Wiki Projects with Smartsheets

NETS pursued these Metropolitan Area Network (MAN) projects in FY2015:

  • Boulder Point-Of-Presence (BPOP)
  • Boulder Research and Administration Network (BRAN)
    • BRAN Table Mesa Flood Mitigation
  • City of Boulder CG4 inter-building cabling

NETS pursued these Wide Area Network (WAN) projects in FY2015:

  • Front Range GigaPoP (FRGP) ongoing management and engineering
    • Expanded Google peering to 20 Gbps
    • Netflix regional caching and upgrade
    • DREAM 20G upgrade
    • FRGP BiSON utilization
    • I2 NET+ support investigation
  • New FRGP participants: I2/USDA
  • City of Denver and USAP departed
  • Internet2
    • Gender Diversity Initiative Co-chair
    • Network/Connector Liaison
  • Bi-State Optical Network (BiSON)
    • Implemented Golden ring
    • Implemented BiSON South
    • UW NSF CC*NIE 100G upgrade
  • XSEDE
  • Western Regional Network (WRN)
    • 100G upgrade implementation
  • NOAA Research Network (NWAVE)
    • Boulder TIC design
    • 910 colocation design and support
    • WRN lambda addendum
  • The Quilt Project – National Regional Networks Consortium
    • Jeff Custard – Executive Committee/Vice-Chair and Secretary
    • Marla Meehl – Nominations Committee
    • Marla Meehl – Finance Committee
    • Marla Meehl – CC*IIE Regional Collaboration Working Group
  • NSF CC*DNI UH Mauna Loa Instrument Proposal to provide high speed wireless networking support

NETS pursued these special projects in FY2015:

  • NSF RMCMOA CC*IIE Grand and WINS Supplement
    • Two RMCMOA workshops
    • Women in IT Networking at Supercomputing (WINS) awardee process
  • Westnet meeting support
    • January 2015
    • June 2015
  • Enterprise Architecture Technical Advisory Board (EA-TAB) participation

NETS will continue to provide support and enhancements for all of these essential networking services. NETS activities are primarily supported through UCAR Communications Pool indirect funds, the FRGP/BiSON, NSF Core funds, and NSF proposal funds.

Detailed project descriptions appear below for three of these projects: Cisco 4510 upgrade and deployment, Aruba Wireless access point upgrade and deployment, and BiSON R7 upgrade.

Cisco 6509 emergency replacement with Cisco 4510s

Unpacking new switches
Rick Mumford and Carlos Rojas-Torres unpack a Cisco 4510 ethernet switch for installation.

UCAR networking depends on Ethernet switches at all locations including the NWSC. These switches provide 1 Gbps network service to all offices, conference rooms, public spaces, etc. and in addition, the switches provide Power over Ethernet (PoE) to power all Cisco IPT phones and many building services like PoE cameras. A little over a year ago, in the process of ordering update Cisco 6509 interface cards, we were notified, without prior warning, that Cisco was no longer supporting PoE on the 6509 platform. This required a rapid redirection of plans to efficiently order and replace all end-of-service hardware in a timely and cost-effective fashion.

Team lead Teresa Shibao worked closely with Cisco to identify and clearly define the problem, evaluate multiple solutions with the network engineering and management team, and worked with management to develop a creative, efficient, and cost-effective solution that has been deftly executed and implemented. The Cisco 4510 was identified as the appropriate technical solution, and Teresa, NETS management, Ellen Martinez, UCAR Contracts, and Dan Wilson, were able to work with Cisco to craft a cost-effective and creative solution (deep discount) and zero interest four-year capital lease (a new and unique solution for UCAR). The capital lease allowed UCAR to fund a multi-year bond equipment expense in one year. This will save UCAR money in the short term with the up-front deep discount and in avoided interest and depreciation expenses. Long-term will also accrue because maintenance for the Cisco 4510 is much less expensive than the 6509 maintenance. It also freed up bond funds for other uses across UCAR.

Emergency replacement team
Led by Teresa Shibao (Network Engineer, not shown), Network Infrastructure Team members Armando Cisneros, Paul Dial, Susan Guastella, Jerome Martinez, Mike Martinez, Rick Mumford, Carlos Rojas-Torres, put in the long early-morning hours and hard work on this project. The team received SCD special recognition awards from Al Kellie, CISL Director, on 20 August. From left to right: Armando Cisneros, Network Technician; Rick Mumford, Network Technician; Al Kellie, CISL Director; Ellen Martinez, Administrator; Paul Dial, Network Engineer; Carlos Rojas-Torres, Network Technician; Susan Guastella, Administrative Assistant. Other team members not shown: Jerome Martinez, Network Technician and Mike Martinez, Network Technician.

Susan Guastella and Ellen Martinez worked deftly through the complicated and new process of ordering under a capital lease, a learning experience for everyone. They set up the capital lease depreciation and four-year management process, again new to everyone involved. They did a splendid job with much patience, creativity, and attention to detail.

Ethernet switch installation
Mike Martinez, Jerome Martinez, and Paul Dial finalizing a 4510 installation.

There were 20 communication closet switches to be installed. Teresa and her team committed to expedite the installation requiring a switch installation every other week, which is an aggressive schedule with extensive planning and prep work for each installation required to ensure success. Teresa and Paul agreed to alternative switch installations as lead to spread the load and provide important technical and leadership experience to Paul Dial.

It was critical that this work get completed within schedule, and the nature of the work requires a level of quality in planning, design, and execution to ensure continuity of the network. The closet switches are the backbone and core of the UCAR network. This work at some point impacted all UCAR network and phone users and the NI Team ensured that the impact was minimal by their excellent quality and execution of their work. The NI Team committed and adhered to an aggressive schedule that demanded that they work outside their normal schedules and perform a magnitude of work that is far beyond normal done in parallel with their regular jobs. The scope had far-reaching impacts that demanded an exceptional commitment to excellence to minimize downtime and impact to users.

Wireless Local Area Network (WLAN) upgrade to Aruba

Access point installation
Rick Mumford installs a new Aruba Access Point.

If there is one technological advancement that has defined the past five years, it is undoubtedly the advent of mobile connectivity. The constraints of connectivity have dissolved in recent years, resulting in an “always on, always connected” society and workplace. The pace of this change has been staggering and its implications are unquantifiable.

In 2015, NETS evaluated various WLAN replacement alternatives to bring our WLAN technology up to the state of the art and better serve the UCAR WLAN users including easier support for the UCAR system administrators who support a myriad of wireless devices and access levels. After multiple demonstrations and cost and technical comparisons, Aruba was chosen as the WLAN vendor.

AirWave management interface   AirWave interface
AirWave is now providing detailed ongoing usage information on over 1,000 wireless client devices connected to the UCAR WLAN.

Aruba ClearPass access management was chosen to offer new options for “onboarding” of wireless clients as well as certificate management. This integrated system from Aruba provides UCAR with several advantages.

First, the AP management system is a “controllerless” model meaning NETS did not have to purchase several pairs of redundant hardware appliances per campus but instead can leverage an “Aruba Instant” configuration that provides deployment flexibility and software-based redundancy using existing AP hardware with no single point of failure.

Second, Aruba offers technology to distribute wireless users across APs and frequency bands, enhanced channel interference management for increased overall WLAN capacity, and protection against failed APs and outside interference sources.

Third, AirWave management provides centralized monitoring and management with network-wide visibility and enhanced correlation and classification options.

Fourth, ClearPass provides a policy platform as well as an improved UCAR wireless user experience due to guest and visitor management portal options.

NETS purchased 802.11ac Aruba Access Points (APs) along with Aruba AirWave network management to control the AP infrastructure.

BiSON R7 Upgrade

WDM installation
John Hernandez, project lead, shows a completed ADVA R7 installed and active.

High Performance Networking (HPN) and the other components of advanced CyberInfrastructure (CI) are key enabling technologies vital to each university, college, and research organization’s ability to function and prosper in a rapidly evolving scientific and technical environment. HPN enables scientific investigations to flourish, along with analysis, communication, global collaborations, education, and outreach missions. Advanced networking infrastructure is an essential enabler of all key elements comprising CI. HPN, commensurate network security, throughput performance, High Performance Computing, data movement, file storage, and retrieval management are all rapidly changing aspects of CI.

Node installation plan
Detailed plan for the installation of the University of Wyoming ADVA R7 ADVA node in Laramie.

The Networking Engineering and Telecommunications Section (NETS) has constructed, operated, and maintained the Bi-State Optical Network (BiSON) for over 10 years. BiSON is the Regional Optical Network (RON) serving Colorado and Wyoming. The BiSON collaboration is an extremely successful partnership among the universities, research, and educational organizations in the region and enables advanced networking in a geographically challenged area (large distances, low population density, and limited State funding). Continued investment in BiSON is critical to meeting the growing and expanding demands of CI in the region.

John Hernandez, Scot Colburn, Pete Siemsen, Bryan Anderson, and Carlos Rojas-Torres were responsible for all design, installation, and documentation of every component of this multi-year project. This included working closely with the optical wave division multiplexing (WDM) equipment vendor ADVA and the constituents on determining unique technical and budgetary requirements and designing the network to meet those requirements. This process requires multiple iterations and technical and negotiation skills. The installation included establishing schedules to minimize interruption and impacts over multiple days and team travel for the multiple physical installations throughout the multi-phase project.

The NETS team has upgraded the full BiSON ring to a state-of-the-art ADVA WDM system that is enabling 100G lambda paths, less costly and more flexible 10G lambda paths, and more flexibility in point-to-point paths. The NETS team worked with UW and the State of Wyoming to install the first 100G lambdas as part of a successful NSF CC*IIE grant enabled by this BiSON upgrade. As part of this grant, UW and the FRGP were able to upgrade the wide area paths serving the FRGP to 100G as well, directly benefitting all FRGP and BiSON participants including UCAR.

BiSON network
BiSON network after upgrade and expansion.

The teamwork on the BiSON upgrade resulted in substantial, innovative achievements in service to the FRGP, BiSON, and UCAR constituents and customers. UCAR directly benefits from this project in our use of BiSON infrastructure supporting scientific research and administrative activities. BiSON is used for UCAR network access intra-UCAR to the NWSC and to the broader Internet for all of UCAR, including all national and international network connectivity. Through BiSON collaboration, considerable cost savings are realized by UCAR and all BiSON participants by leveraging the economies of scale realized by shared resources.

Fiber optic paths and WDM equipment are very costly to acquire and maintain. By maximizing the relationships and purchasing power of 10 major R&E organizations in the region, we have all been able to do more with less – higher speed, more flexibility, more geography, more grant funding, and more collaboration within the region and the broader Internet. This, in turn, enables more science, education, and research.

Through the FRGP and BiSON, regional and national collaboration and positive exposure for UCAR are also realized. An example of regional collaboration enabled and supported directly by BiSON is the Rocky Mountain Advanced Computing Consortium (RMACC). HPC across the region utilizes BiSON to share computing and storage resources. The Quilt and Internet2 are examples of national collaborations where UCAR/FRGP/BiSON participate and UCAR benefits directly from the national exposure of technical and regional leadership.