Spending days and nights monitoring your own network, as well as others, would probably seem like a mundane task to majority of people, but like any other job there are certain days that can stick out in your mind.
For the ones that are not familiar with what a Network Operations Center or NOC is, it is one or more locations that does network management, network monitoring and controlling over a computer. They watch your computer networks to see if there is any issues, threats, or attacks occurring and respond to them quickly. This is typically depicted in images with multiple desks in a room, copious amounts of coffee, a minimum of two monitors per seat, and larger monitors mounted to the wall. Each monitor displaying graphics and lists with varying colors; red typically highlighting high priority alerts.
NOC engineers review and scan loads of data throughout the day and sometimes night, quickly picking out what is a possibly problem before it becomes catastrophic. Although this can be at times monotonous there are days that are spent putting out fires or sometimes a blaze. Check out the stories below of some Untold Stories of the NOC from current and past NOC engineers.
Wrong Number?
“I once had a frontline TAC (Technical Assistance Center) engineer answer the phone by saying ‘thanks for choosing Wendy’s.’ He then got all flustered and said “jeez I haven’t worked there in years.” It made light of an otherwise tense situation.”
Are We On Mute?
“I am on the phone with TAC where I hear the supervisor in the background say ‘no, you shouldn’t have deleted that folder. Now take the customer off mute and tell them we have to restore from backup because the database is corrupted.’ Lucky for us we had full backups and were able to resort fine.”
Have You Seen The Latest Thriller?
“During an overnight shift in the NOC I once noticed a quick, yet large spike in traffic, coming inbound to the external address of one of our clients. This at the time was normal for the particular client due to the nature of their business in ecommerce. Upon further investigation we started digging into the netflow logs and noticed subsequent smaller sessions leaving their network to random addresses across the globe. The connections were coming from a server in their network, which had no reason to communicate externally and we became very suspicious. After digging into the firewall logs we determined the ports, IPs and data being sent to these locations. The server was recently compromised and configured as a warez drop to host pirated movies and software. We contacted the customer, isolated the threat and assisted with the cleanup of the server configuration. All in a nights work.”
Did You Patch It?
“Not one day particular, but I guess one of the most Famous TAC response I would get a lot is ‘I’ve seen this before, just install this patch.’”
Are The Bills Paid?
“We received a call once that everything on our customers side was down. We also could not see any flow on our monitors. All said and done we found out that the customer didn’t pay their service bill. We ended up finding out, had to call the customer back, and explain the service isn’t working because you haven’t paid your bill.”
How about you? Do you have any funny or interesting NOC stories?
Author Bio: The CCSI Team, for more than 40 years, Contemporary Computer Services Inc (CCSI) has provided clients in both the private and public sectors with a rock solid foundation on which to secure their organization’s future. The CCSI NOC/SOC is a 24×7 operation center that focuses on the uptime, availability, and operations for our clients. Our certified engineers are monitoring systems around the clock and are the extension to your operations team.
You can reach out to the CCSI team on Twitter, Facebook, LinkedIn, or via our contact us page.
