Calendar server outage

Calendar Server

8:00am — InfoSys noticed that the calendar server was not responding to connection requests. Diagnosis and repair is ongoing.

10:30am — Info Systems has determined that a major failure within the Linux system occurred, probably sometime before 10pm Tue night. A Network Admin has opened a trouble ticket with Red Hat and several phone conversations with various levels of tech support have occurred. Current level is with a senior systems tech. Because of the nature of this failure we want to be able to determine the root cause before doing any drastic restore to the Linux system software. The Oracle calendar data appears, at this point, to be safe and intact. We have no projected time for the server to be back online.

11:30am — Info Systems confirms that the Oracle Calendar server has been hacked. The system components (other than the oracle application and database) will be restored from the system image backup.

1:00pm — Validation of the Oracle Calendar database fails. It appears that significant corruption of the data has occurred. A restore of the database from the backup of 4:00am, Tues, 1/6/04 will be performed.

2:30pm — Restore of Oracle Calendar database is complete and appears to be valid. Jeremy Good will send an email to all current Oracle Calendar users with details of what has occurred and the state of their data as of now. Any items entered or updated in Oracle Calendar after 4:00am Tues, 1/06, have been lost. It is possible that persons who have PDA devices that they synchronized after 4:00am Tues may be able to load these items from the PDA back into the Oracle Calendar. However, it will be necessary to contact Jeremy to confirm the proper procedure to use to do this.

NOTE: Although the Oracle Calendar server has been returned to service Info Systems must now load a Red Hat Linux kernel patch to fix the vulnerability that was exploited by the hacker who caused this damage. This process will be done after 5:00pm today (Wed, 1/7). Until that is done all access to the Oracle Calendar server outside the EMU firewall will be closed and will only be opened after confirming the system is properly functioning on Thu morning, 1/8. An entry to that effect will be added to this outage log on Thu morning.

POSTPONED: ReConfigure Internet Connection for Increased Capacity

Internet

POSTPONED: The following procedure, originally scheduled for 12/16/03, will be postponed until January due to unavailability of services from our Internet supplier until then.

J.Rutt — 12/12/03
———————–
The EMU Internet connection will be switched from the current configuration of two DS1 lines that provide 3 megabits per second (mbps) capacity to a single ethernet connection of 4mbps. This new connection provides higher capacity (can be increased up to 10mbps) for a lower cost per megabit.

This change has been prompted by the project to implement Video TeleConferencing (VTC) which is being made possible by the LEAP Lilly grant for the seminary.

Network administrators will be working with consultants from STG of Winchester to make this conversion. It is estimated that about 1 hour of actual down-time will be required.

During the outage all access to the Internet will be disrupted. The email servers will remain operational but all incoming and outgoing mail will be held in various queues.

Persons outside EMU will NOT be able to get to the EMU web servers (EMU Home page, Blackboard, WebMail, WebCalendar).

ST rebooted

File Server

The Novell server “ST” was rebooted this morning due to a stuck process. This process caused the CPU to constantly stay at around 30%utilization.

The backup had hung because of this problem.

In trying to figure out what/why I noticed that a users connection could not be disconnected.

So are we chasing a server or client problem?