Calendar server was not responding. First noticed about 8am, Sunday 2/20. Calendar administrator verified that a reboot was required. Server rebooted at 9am. Not sure of exact time the failure occured. was sometime between mid-night and 8am.
Category Archives: Calendar Server
Calendar: server unavailable
The calendar server was unresponsive via the network or the console. A “spurious interrupt” message appeared on the console. After a reboot, everything appears fine.
Calendar: server unavailable
The server needed a reboot and database rebuild. The server was running, but was unresponsive. Disk repair (fsck) was performed at boot.
Oracle Calendar server unresponsive
The server was running, but it was not possible to log in as administrator. After a “hard” reset, fsck ran and corrected file system errors. Ran calendar database repairs and then started the calendar server.
Calendar: Server is non-responsive
The system was not available for user or administrative access, and was restarted. File system and database repair utilities were run.
Calendar outage.
The web component of the calendar server was offline for unknown reasons. When restarted the server reported no errors.
Calendar server maintenance and diagnostics
Routine maintenance was performed, as well as gathering of diagnostic information for Oracle support.
Oracle Calendar Server Database Problems
There were ongoing problems with the Oracle Calendar Server, particularly for Outlook Connector users. Some users could not see some of their folders (Calendar, Tasks, Contacts, Notes) while using Outlook Connector. There also appeared to be a problem with creating recurring calendar events using Outlook Connector.
It does not seem likely that these problems are related to the Oracle Calendar outage of 1/7/2004.
Oracle provided assistance to run advanced database repair tools and correct the problem. Calendar users are advised to review their data to check for inconsistencies.
Linux Kernel Security Upgrade on Calendar Server
An upgrade to the Linux kernel will be performed on the Calendar server following its being rebuilt today as a result of a hacking/cracking incident. Internet access to the Calendar server has been closed since the rebuild brought it online at 2:30pm today (1/7). It will remain closed to Internet access until about noon on Thursday, 1/8.
Calendar server outage
8:00am — InfoSys noticed that the calendar server was not responding to connection requests. Diagnosis and repair is ongoing.
10:30am — Info Systems has determined that a major failure within the Linux system occurred, probably sometime before 10pm Tue night. A Network Admin has opened a trouble ticket with Red Hat and several phone conversations with various levels of tech support have occurred. Current level is with a senior systems tech. Because of the nature of this failure we want to be able to determine the root cause before doing any drastic restore to the Linux system software. The Oracle calendar data appears, at this point, to be safe and intact. We have no projected time for the server to be back online.
11:30am — Info Systems confirms that the Oracle Calendar server has been hacked. The system components (other than the oracle application and database) will be restored from the system image backup.
1:00pm — Validation of the Oracle Calendar database fails. It appears that significant corruption of the data has occurred. A restore of the database from the backup of 4:00am, Tues, 1/6/04 will be performed.
2:30pm — Restore of Oracle Calendar database is complete and appears to be valid. Jeremy Good will send an email to all current Oracle Calendar users with details of what has occurred and the state of their data as of now. Any items entered or updated in Oracle Calendar after 4:00am Tues, 1/06, have been lost. It is possible that persons who have PDA devices that they synchronized after 4:00am Tues may be able to load these items from the PDA back into the Oracle Calendar. However, it will be necessary to contact Jeremy to confirm the proper procedure to use to do this.
NOTE: Although the Oracle Calendar server has been returned to service Info Systems must now load a Red Hat Linux kernel patch to fix the vulnerability that was exploited by the hacker who caused this damage. This process will be done after 5:00pm today (Wed, 1/7). Until that is done all access to the Oracle Calendar server outside the EMU firewall will be closed and will only be opened after confirming the system is properly functioning on Thu morning, 1/8. An entry to that effect will be added to this outage log on Thu morning.