Monthly Archives: January 2004

Oracle Calendar Server Database Problems

Calendar Server

There were ongoing problems with the Oracle Calendar Server, particularly for Outlook Connector users. Some users could not see some of their folders (Calendar, Tasks, Contacts, Notes) while using Outlook Connector. There also appeared to be a problem with creating recurring calendar events using Outlook Connector.

It does not seem likely that these problems are related to the Oracle Calendar outage of 1/7/2004.

Oracle provided assistance to run advanced database repair tools and correct the problem. Calendar users are advised to review their data to check for inconsistencies.

Calendar server outage

Calendar Server

8:00am — InfoSys noticed that the calendar server was not responding to connection requests. Diagnosis and repair is ongoing.

10:30am — Info Systems has determined that a major failure within the Linux system occurred, probably sometime before 10pm Tue night. A Network Admin has opened a trouble ticket with Red Hat and several phone conversations with various levels of tech support have occurred. Current level is with a senior systems tech. Because of the nature of this failure we want to be able to determine the root cause before doing any drastic restore to the Linux system software. The Oracle calendar data appears, at this point, to be safe and intact. We have no projected time for the server to be back online.

11:30am — Info Systems confirms that the Oracle Calendar server has been hacked. The system components (other than the oracle application and database) will be restored from the system image backup.

1:00pm — Validation of the Oracle Calendar database fails. It appears that significant corruption of the data has occurred. A restore of the database from the backup of 4:00am, Tues, 1/6/04 will be performed.

2:30pm — Restore of Oracle Calendar database is complete and appears to be valid. Jeremy Good will send an email to all current Oracle Calendar users with details of what has occurred and the state of their data as of now. Any items entered or updated in Oracle Calendar after 4:00am Tues, 1/06, have been lost. It is possible that persons who have PDA devices that they synchronized after 4:00am Tues may be able to load these items from the PDA back into the Oracle Calendar. However, it will be necessary to contact Jeremy to confirm the proper procedure to use to do this.

NOTE: Although the Oracle Calendar server has been returned to service Info Systems must now load a Red Hat Linux kernel patch to fix the vulnerability that was exploited by the hacker who caused this damage. This process will be done after 5:00pm today (Wed, 1/7). Until that is done all access to the Oracle Calendar server outside the EMU firewall will be closed and will only be opened after confirming the system is properly functioning on Thu morning, 1/8. An entry to that effect will be added to this outage log on Thu morning.