Last day to save $500 for SANS San Diego 2013

Doug Brown, former Manager of Security Resources, University of North Carolina at Chapel Hill

Introduction


Table of Contents

Doug Brown, former Manager of Security Resources, University of North Carolina at Chapel Hill

October 30th, 2008
By Stephen Northcutt


One of the important concepts that we want to explore in security thought leadership is the idea of group or team thought leadership. And so we are looking for examples of teams that exhibited security thought leadership. Doug Brown, former Manager of Security Resources, University of North Carolina at Chapel Hill, was on a team that exhibits many of the characteristics of security thought leadership.


Doug, can you introduce us to the team that you served on?

Thank you, Stephen. From 2001 until 2006, the University of North Carolina at Chapel Hill had a strong team of collaborative players in their Information Security and Networking groups. These were the early, wild-west years for much of what has now become Information Security canon. Inventing as they worked through each day’s new challenges, this team happily shared their successes with colleagues and also the few vendors who would listen.


Did this team have a Code or Guiding Principles?

As a matter of fact, we did; a few of our main guiding principles were:
  • Protect the University, protect the data, and protect the systems.
  • Security that is onerous or cumbersome is security that gets circumvented: always balance risk, efficacy, and transparency.
  • Clearly and frequently communicate the case, build consensus with the customers, and move forward to advance the cause.
  • Not having all the answers is not a mission ender; leverage all the available resources and friends, because someone else has already likely solved at least part of your problem.
  • Education and proactive measures provide the best ROI.

As I understand it, UNC was one of the early adopters, like VA Tech, of using VLANS to increase Security and manage systems on the network. I believe this was called SecureFast, is that right?

In 2001 as UNC's Y2K office transitioned into their Information Security office, an essential relationship of cooperation already existed between the Security and Networking teams. Prior to a subsequent migration to 802.1q, the University's network relied on the proprietary VLAN architecture of Enterasys' SecureFast. Security awareness was central to Enterasys well before any other networking vendors. The SecureFast VLAN technology allowed UNC to implement two fundamental tools to support their early security efforts; these were referred to internally as the Penalty Box and Source Blocking.

SecureFast based VLAN membership on a device's MAC address rather than its current switch port. This simple concept allowed UNC's Networking and Security teams to quickly move compromised or infected systems into an isolation VLAN. This VLAN did not allow any internal or external communications: nefarious systems could no longer speak to each other or any one else. It was often the case that the student or departmental administrator would try a different wall port; however, the Penalty Box was tied to the system's MAC address and any number of ports tried would yield the same lack of communication.

Jim Gogan was a big hockey fan and coined the name Penalty Box. Borrowing from the movie Alien we often said "In the Penalty Box, no one can hear you scream." Occasionally an ambitious administrator would attempt replacing network cards, sometimes going through several, and then other safeguards would kick in to stop bad traffic and provide protection.


Can you tell us more about the drawbacks of the Penalty Box?

Placing systems in the Penalty Box was originally a manual process. The Security office would identify the bad actors and refer them to the Networking group for isolation. As the size and speed of security events began to increase, one commonality became apparent: systems attempting to infect others would generate insanely high numbers of unresolved ARP requests as they scanned empty IP space for new victims. SecureFast tracked these requests and UNC's team was able to add an automated dimension to their security tools by implementing Source Blocking. Those systems that crossed the threshold for a reasonable amount of unresolved ARP requests were cut off from communicating. This also had the benefit of catching those engaged in inappropriate experiments with security tools.


UNC was one of the early schools that had a laptop requirement during this time. Can you tell us a bit about that helped security?

UNC implemented an educational program requiring incoming freshmen to have laptops meeting a minimum specification. This Carolina Computing Initiative (or CCI) offered IBM ThinkPads with a University-provided software load as one possible source for students to meet the requirement. More then 90% of the students purchased these IBM systems, and the opportunity to provide students a system image that was both supportable and secure out of the box proved invaluable to both the Help Desk and Security teams. This system image was reviewed and tweaked twice each year by a committee of different interest groups. Small changes, such as the addition of a centrally managed anti-virus client and the shut down of unnecessary services, meant that the student networks exceeded the availability and performance seen at similar institutions that were not providing a desirable system image.


What did you do to educate the University population about computer security?

Security training is essential; however, it’s also expensive and problematic on a limited budget. Providing the foundation to do things correctly, or securely, from the beginning meant fewer problems after administrators moved systems into production. To build these foundations on the cheap, the UNC Security team developed internal programs to provide campus administrators a hands-on opportunity to build and secure both Windows and RedHat server systems. These day-long classes equipped UNC administrators with secure build experience and resulted in repeatable results when they returned to their departments.

Sensing the desire and need for broader security training, the UNC Security team partnered with the SANS Institute and through the local mentor program I offered several sessions of Security Essentials to the campus community. This in depth security training, offered at a group discount and sparing the expense of travel, allowed the UNC Security team to build a cadre of security advocates and incident response experts in various departments around the University. Having resources on the ground with the requisite skills to both prevent and contain security issues multiplied the manpower of the limited Security team staff.

The UNC Security team saw how other universities were struggling with information security and presented at Educause on the various free security tools that were available at the time. Providing an end-to-end roadmap from prevention, to detection, to clean up, this look into how the sausage was made gave the attendants at that year's conference an opportunity to kick off their own security efforts in the absence of any real budget priority.

For the student population, we realized that we were probably too old and square to really connect with the right message. We engaged, and paid, some students who were interested in video production to craft some commercial-like messages about information security topics. We provided the list of topics and allowed them to run with their ideas. The results ended up being pretty catchy.


And to make your joy complete, I understand that the team also tackled some major wireless issues. How did that work out?

For campus networks that were early adopters of both Wireless and routed VLANs, Microsoft's network bridge, initially enabled by default in XP Home, was a disaster. Did Microsoft understand networking. Hadn't they ever heard of spanning tree loops. Trying to work with Microsoft was like yelling at a wall. Fortunately, most of the systems coming on to UNC's network included UNC's image and the configuration could be made right; however, the occasional system from outside the program could wreck havoc on portions of the campus backbone.

What was once a little known feature buried in some vendors' switch code eliminated this problem at UNC. BPDU blocks, or Span Guard, killed the wired port for any system seen attempting to create a bridge. This cleared up the network loops, but wireless management still left a lot to be desired. The penalty box and source blocking tools available on the Enterasys switch ports did not extended to the Cisco wireless access points in use at the time, and the movement to 802.1q VLANs meant the SecureFast tools were going away.

Repeated requests to the vendor for a method to drop infected systems from wireless were met with no reply. It seemed the sales engineer and his technical resources had never heard of such a request and really had no interest in finding a workable answer. The UNC team took it upon themselves to crack the puzzle and discovered that, since an AP is a bridge in disguise, Cisco's set of bridge commands (including discard) worked on their APs. While SNMP was generally the preferred way to manage equipment, the only avenue available on those early APs was management through telnet. Accordingly an "expect" script with the necessary telnet commands was written to facilitate the speedy removal of bad actors from the wireless network.

UNC's Security team involved themselves in various outreach efforts, including the leadership behind a well-received white paper for implementing secure wireless in healthcare, and extensive contributions to the development of an Information Security Baseline for the 16-campus system of the State of North Carolina. An outside vendor was later engaged to audit each campus against both this system baseline and ISO 17799. For such a large University system, North Carolina was ahead of the times.


Looking back, what advice do you have for anyone considering a major wireless rollout?

Evaluate all the equipment that you can get your hands on and really dig into their management interface. How quickly can you find a particular user’s system. How quickly can it be dropped or isolated. Is their management system standards based. Will it interoperate with other best-of-breed solutions like TippingPoint?


Can you share some more about the technological tools that helped you manage such a large and amazing network?

As more and more new Microsoft vulnerabilities were discovered, the UNC Security and Networking teams continued to see the size and scale of security events increase at an exponential pace. Passive intrusion detection, like our much beloved Snort, had become a postmortem tool. Manual efforts to identify and isolate infected systems lost their value as hundreds of hosts fell victim to Blaster and other RPC vulnerabilities. By now UNC's network had fully moved to 802.1q VLANs, and many of the old SecureFast tools were gone.

Early success as the second customer of TippingPoint showed both the value of Intrusion Prevention and the uniqueness of TippingPoint's approach. UNC's Security and Networking teams soon recognized the opportunities of this powerful tool, and beyond their sub-division of the campus network into multiple "attack domains," the UNC team home grew the pieces to combine TippingPoint's IPS with their powerful network management tools from Enterasys.


We are always interested in tools, do you know if they are still running TippingPoint and Enterasys today? Any thoughts about these tools for someone implementing a network today?

As I understand it, the core of what we built is still in use and the Networking group is focused on a process of constant improvement. When you find good vendors who listen to customers, and they quickly implement requested changes, then the opportunities to really make some gains open up for everyone. Vendors who maintain the flexibility to fit your needs will sell plenty of product.


Thank you for that, Doug; so, you were able to achieve sanity through the proper use of technology?

All these pieces led to a completely automated, standards-based, vendor-neutral, response to several IPS signatures that were known to be 100% accurate to identify infected systems. Through various methods this environment grew to the automated moving systems to a limited self-help VLAN, the forcing a DHCP release and renew regardless of lease times, and the creation of a detailed Remedy ticket so the Help Desk could know immediately why a system had been isolated. The UNC Help Desk often contacted the user before they realized there was a problem.


And I understand that the world noticed your achievement, can you tell us about that?

The Information Security environment at UNC remained a transparent part of an open network meant to encourage and support education and research. No pre-authorization checks stood in the way of systems joining the network, and yet infected systems were removed with enough speed that their bad traffic could not impact others. The self-help VLAN provided access to tools like fresh anti-virus definitions, Windows updates, and a security scan by the Whole Security Confidence Online behavioral anomaly detector. Through creative subnetting, systems in self-help could not attack each other, or anyone else, but they could access some rudimentary tools to get them on the road to recovery.

UNC’s extensive program, ranging from their secure system images, to their education efforts, their excellent network management, and their willingness to become early adopters and supporters of highly effective new technologies such as TippingPoint's IPS, made the group of people accomplishing things at UNC a team of Information Security thought leaders. They were recognized in 2005 as Network World All-Stars for their efforts to automate security response.[1]


Doug, I really appreciate your taking the time to share, this group clearly achieved thought leadership, who were the members of this team?

Many folks contributed, but the core members of this successful UNC Security and Networking team were:
  • Jeanne Smythe, CISM
  • Jeff Bollinger, GSEC, CISSP
  • Doug Brown, GSEC, CISSP
  • Jim Gogan
  • Mike Hawkins
  • Joni Keller, Phd.
  • Chuck Crews
  • Keith Makuck

Can you tell us anything about the team dynamics. Can you talk about a time you were under extreme stress, maybe when blaster hit. Who took a leadership role?

We happy band of brothers (and sisters) grew to know each other well enough to complete each others’ sentences. During crisis or extreme stress the team member with the current answer took the leadership role, until the answer changed or was further refined. We were always focused on solving the problem; personalities rarely got in the way, especially when an argument with technical merit was to be had. We kept meetings small and short. Sometimes quick decisions had to be made, but even those usually had the quick and concise input of three or more people - often a sanity check: dude, are you seeing what I’m seeing?

One official once commented that "those guys from Security roll their own"; we weren’t rolling anything, and weren’t entirely sure what he meant, but we took it as a compliment anyway.


Thanks, can you talk about a time the team was divided on an issue. How did you sort that out and develop consensus?

There were times when more then one solution would solve the problem; when that happened it often became a divide and conquer situation of "you try your way with that one and I’ll try my way with this one." We’d then circle back and come to consensus on what was cleanest, quickest, and best for the customers.


Do you still think of yourself as a team-oriented person or does your current job position cause you to be in more of an individual contributor role?

I consider myself to still be very team oriented; a new team with new challenges but still very much a team that works together to find what is best for the customers. I firmly believe that a strong team will almost always out-perform any individual.


If there was one thing you really wanted to share with our readers, a rant, a tip, an observation, even a charge to action, what would that be?

UNC’s success came from cherry picking the best-of-breed solutions to fit each need. Do not buy into the marketing hype that any single vendor can supply all of your security needs, you’ll need to work with several different security vendors.


Thanks so much Doug, can I ask you to tell us a bit about yourself, what do you do when you are not in front of a computer?

When I’m not spending time with my wife and kids, I’m usually studying for my graduate classes or tinkering with non-electronic things. On occasion I’ll dress for 1776 and join a group of folks who reenact the Department of the Army Geographer from the American Revolution. This group of hobbyists focuses on the technologies of surveying and cartography from the days before GPS made our lives so easy. It’s a great way to get away from the machines for awhile - sleeping in canvas tents around a campfire and cooking salt pork.

====
1 http://www.networkworld.com/allstar/2005/112105-education-more.html?page=3
<< Thought Leader Home
top^
<< Thought Leader Home

SANS CDI 2013 - DC

Latest Whitepapers

Building and Maintaining a &quot;Certifiable&quot; Workforce
By Robert J. Mavretich

How Can You Build and Leverage SNORT IDS Metrics to Reduce Risk?
By Tim Proffitt

Daisy Chain Authentication
By Courtney Imbert

Latest Tweets

NEW #SANS Survey: Security Analytics & Intelligence 2nd [...]
October 1, 2013 - 6:30 PM

Tips on how to convince your boss to let you train online wi [...]
October 1, 2013 - 6:23 PM

#2 Tip on how 2 convince your boss 2 let u train online: Fle [...]
October 1, 2013 - 3:00 PM

Contact Us

(301) 654-SANS (7267)
Mon-Fri 9am - 8pm EST/EDT
info@sans.org

"As a security professional, this info is foundational to do a competent job, let alone be successful."
- Michael Foster, Providence Health & Security

"It was a great learning experience that helped open my eyes wider. The instructor's knowledge was fantastic."
- Manuja Wikesekera, Melbourne Cricket Club

"The amount of knowledge conveyed and technical diving by practical hands-on was well balanced."
- Aaron Moore, Maricopa County