Network Outage - March 11th
March 12, 2015
As many of you are aware, we experienced an internet outage on Wednesday evening, March 11th. The incident started at 4:54pm and ended at 7:37pm. While it is our intention to provide the best possible service, we apologize for the disruption this incident caused our community. We all rely heavily upon the college’s network and although it runs steadily 99.95% of the time, when it goes down it’s a big deal to all of us.
The issue had to do with an advanced routing protocol on our firewall, which failed due to a configuration setting, resulting in network traffic being rejected. This outage was very similar in nature to the internet outage that occurred on February 13th. Finding the root cause of these sort of issues can be a bit like searching for a needle in a haystack, but fortunately in this case it was found and traced back to a configuration error.
So why was there a configuration error? Well, as part of an ongoing process of managing our infrastructure, the firewalls were recently replaced. To assist with this work, we hired outside professional services that know these systems better than we do. Our expectation is that these services help us configure the more advanced settings, but in this case a piece was missed. To try and keep this from happening again, we are engaging an engineer from our firewall vendor to conduct a thorough audit of our implementation in order to discover if there may be other potential issues at play that also need to be resolved.
Our goal is to have the most reliable network possible on campus. Does that mean it will never go down again? We certainly hope not, but in a real world, it could. Our commitment to the college community is that we will continue to work hard to provide a speedy and reliable network that you can depend on.
If you are interested in learning the specifics about this (and other) outage, please visit our Outages and Updates page.
If you have any questions or concerns, please do not hesitate to contact us at the IT Service Desk, firstname.lastname@example.org or x7225.
For more information:Information Technology Service Desk