Outages and Updates
Incident: FTP Server Down, April 13, 2015
ftp.lclark.edu is function again on a new Solaric VM. We have added the users that were earlier identified as needing accounts to the etc/password file, so they are the only people who can log in. These logins were tested and it was verified that they could access their cwis directories.
The bad news is that this server doesn’t not authenticate via LDAP, so these users have a different password for this server. If you need access to update your webpages on legacy.lclark.edu and have not yet been contacted by Information Technology, please contact the IT Service Desk at email@example.com.
(Updated April 17, 2015)
The SunFire V480 that hosts ftp://ftp.lclark.edu has been corrupted directly affecting FTP access to http://legacy.lclark.edu/ files. This outage only affects those users trying to edit their sites hosted at http://legacy.lclark.edu/~username. Another server serves up the files so sites are still viewable. We hope to have this server functional on Wednesday, 4/15/15.
(Updated April 14, 2015)
Incident: Internet Outage, March 11, 2015 - 4:54pm to 7:37pm
The access between our trust network (internal) and our untrusted network (outside) was disrupted. The normal route to the Internet that goes through our firewall stopped routing properly. Due to a configuration error with OSPF adjacency, the firewall rejected the default route provided by our Internet border gateways thus resulting in a failure to be able to route traffic to the Internet.
As a normal course of action, we contacted our firewall vendor and engaged their technical help in order to discover the cause of the issue. Once the technician determined the issue, a change was made to the firewall in order to accept the default route from our border gateways. This change resolved the routing issue immediately and Internet service was restored.
A thorough audit is planned of our recent firewall implementation so we can make sure there are no other issues looming that could potentially affect the stability of our network.
We appreciate your patience and understanding as we work through some of the recent hiccups. We are hopeful that yesterday’s outage will be the last network disruption in the foreseeable future.
(Updated March 13, 2015)
Incident: Internet Outage, March 11, 2015 - 4:50pm to 8:15pm
The access between our Trust network and our Untrusted network was disrupted. The normal route to the internet that goes through our Firewall was not routing properly. Due to a configuration error, the security appliance rejected the default route provided by our border gateways. It took time as we worked with our Firewall vendor to discover the cause of the issue. Once that was determined, the security appliance was configured to accept the default route from our border gateways and the problem was resolved, bringing our internet back up and functional.
(Updated March 12, 2015)
Incident: Internet Outage, February 13, 2015 - 2:48pm to 3:50pm
So far, the root cause looks to have been our new Palo Alto firewalls choked up on OSPF neighboring. They lost communication to the border routers for some unknown reason. A reboot of the primary firewall is what resolved the outage.
We tried to send log files to the ISO preferred vendor, Fishnet (now Accuvant), and their upload site was down on Friday. We were able to send the logs this morning and are waiting to see if they can determine why the OSPF neighboring failed. I guess it’s important to note that this was not a human introduced outage…meaning there was nobody making any changes (at least nobody on the Infra team).
(Updated February 16, 2015)
Scheduled Outage: Wi-Fi, February 24, 2015
Information Technology will be performing maintenance on Lewis & Clark’s Wi-Fi network on Tuesday, February 24th, from 6AM – 8AM.
During this time, wireless connectivity will be disrupted including wireless connection to the internet. Service will be intermittent during this time. This includes both LC Wireless and LC Secure.
This service disruption will affect Lewis & Clark wireless network services only. Connections to the network/internet via Ethernet cables will still function, unless something unforeseen arises.
(Updated February 10, 2015)