The Student Information System (iSeries, Campus Web) will be unavailable while routine PTFs (Program Temporary Fixes) are applied to the system. A two to three hour interuption is anticipated.
The systems affected will be Jenzabar application software (TEAMS2000) and CampusWeb (https://campusweb.emu.edu). CampusWeb provides student access to their financial information (billings and FinAid) as well as final class grades.
No other systems are affected by this outage.
NOTE #1: Last updated 01/20/04 11:30am The most recent updates are in BOLD text below.
NOTE #2: All available work-arounds have been completed. Marketing Services (Communications Dept) and Development Offices will remain without network connectivity until the permanent fix is installed sometime on Tuesday, 1/20.
A Cisco fiber switch located in the Library failed completely about noon today (1/17). This is one of two major fiber switches on campus, providing network service to all the buildings in the southeast corner of campus.
Info Systems is aware of the problem and is working to obtain a replacement device. It is not clear how long this will take but it is unlikely to be before Tuesday, 1/20.
Network connectivity to the following buildings is affected:
Library
Science Center
Marketing Services (Communications)
Development
Martin House
CTP
Weaver House
Only connectivity to these buildings is affected. All servers remain functional (i.e. Internet, email, calendar, SIS(As400), web, Blackboard, Novell). You just can’t get to them from these buildings.
A Critical Information Alert has been posted and will be updated, along with this log, if/when the status of this outage changes. We apologize for the major inconvenience this places on the campus and assure you that we are working as quickly as possible to resolve the problem.
01/20/04 11:10am Replacement part arrived. About an hour of work will be required to prepare it for installation. We expect to install it between noon and 1:00pm. About a 15-20 minute outage in the affected buildings will be required. A 10 minute “warning” will be sent as a Novell Popup Message that will appear on all Windows computers logged into Novell.
01/19/04 10:00am Successful work-around found for the Science Center, Martin House, CTP, Brunk House, Weaver House. All network connectivity to these buildings has been restored as of 10:00am.
01/19/04 08:00am Successful work-around found for the Library. All network connectivity to the Library has been restored as of 8:00am.
01/18/04 03:45pm Cicso confirms via email that the replacement 3508 switch has been prepared for shipment to EMU. It is likely that it will arrive Tue morning, 1/20.
01/17/04 04:00pm TAC opened with Cisco. They will send replacement hardware. Will be to FedEx by Monday and to us by Tuesday. Likely by 11am, but contract only requires by 4pm.
01/17/04 03:00pm Cause found and notice of failure published on Outage Log, Critical Info Alert and Special Notices on Connection page.
01/17/04 02:15pm Problem first investigated by Info Systems
A new version of the TMA software is available and TMA recommends that all customers upgrade. TMA will be unaccessible through the desktop client and web interfaces during this time.
The mail server, web server, and several other core servers will be taken off line for system maintenance. Individual servers may be available during this time, but this is not assured and should not be relied upon.
There were ongoing problems with the Oracle Calendar Server, particularly for Outlook Connector users. Some users could not see some of their folders (Calendar, Tasks, Contacts, Notes) while using Outlook Connector. There also appeared to be a problem with creating recurring calendar events using Outlook Connector.
It does not seem likely that these problems are related to the Oracle Calendar outage of 1/7/2004.
Oracle provided assistance to run advanced database repair tools and correct the problem. Calendar users are advised to review their data to check for inconsistencies.
We plan to install some critical updates to the webmail system. During the outage users will not be able to access webmail, though the email system will be otherwise unaffected and users accessing their email with desktop clients will not be affected.
An upgrade to the Linux kernel will be performed on the Calendar server following its being rebuilt today as a result of a hacking/cracking incident. Internet access to the Calendar server has been closed since the rebuild brought it online at 2:30pm today (1/7). It will remain closed to Internet access until about noon on Thursday, 1/8.
Email messages are not being delievered in a timely manner. Messages are not lost, simply delayed.
8:00am — InfoSys noticed that the calendar server was not responding to connection requests. Diagnosis and repair is ongoing.
10:30am — Info Systems has determined that a major failure within the Linux system occurred, probably sometime before 10pm Tue night. A Network Admin has opened a trouble ticket with Red Hat and several phone conversations with various levels of tech support have occurred. Current level is with a senior systems tech. Because of the nature of this failure we want to be able to determine the root cause before doing any drastic restore to the Linux system software. The Oracle calendar data appears, at this point, to be safe and intact. We have no projected time for the server to be back online.
11:30am — Info Systems confirms that the Oracle Calendar server has been hacked. The system components (other than the oracle application and database) will be restored from the system image backup.
1:00pm — Validation of the Oracle Calendar database fails. It appears that significant corruption of the data has occurred. A restore of the database from the backup of 4:00am, Tues, 1/6/04 will be performed.
2:30pm — Restore of Oracle Calendar database is complete and appears to be valid. Jeremy Good will send an email to all current Oracle Calendar users with details of what has occurred and the state of their data as of now. Any items entered or updated in Oracle Calendar after 4:00am Tues, 1/06, have been lost. It is possible that persons who have PDA devices that they synchronized after 4:00am Tues may be able to load these items from the PDA back into the Oracle Calendar. However, it will be necessary to contact Jeremy to confirm the proper procedure to use to do this.
NOTE: Although the Oracle Calendar server has been returned to service Info Systems must now load a Red Hat Linux kernel patch to fix the vulnerability that was exploited by the hacker who caused this damage. This process will be done after 5:00pm today (Wed, 1/7). Until that is done all access to the Oracle Calendar server outside the EMU firewall will be closed and will only be opened after confirming the system is properly functioning on Thu morning, 1/8. An entry to that effect will be added to this outage log on Thu morning.
A service pack was installed which fixes several critical issues with Blackboard.