All posts by admin

MAJOR NETWORK FAILURE – South End of Campus

Network

NOTE #1: Last updated 01/20/04 11:30am The most recent updates are in BOLD text below.

NOTE #2: All available work-arounds have been completed. Marketing Services (Communications Dept) and Development Offices will remain without network connectivity until the permanent fix is installed sometime on Tuesday, 1/20.

A Cisco fiber switch located in the Library failed completely about noon today (1/17). This is one of two major fiber switches on campus, providing network service to all the buildings in the southeast corner of campus.

Info Systems is aware of the problem and is working to obtain a replacement device. It is not clear how long this will take but it is unlikely to be before Tuesday, 1/20.

Network connectivity to the following buildings is affected:

Library
Science Center
Marketing Services (Communications)
Development
Martin House
CTP
Weaver House

Only connectivity to these buildings is affected. All servers remain functional (i.e. Internet, email, calendar, SIS(As400), web, Blackboard, Novell). You just can’t get to them from these buildings.

A Critical Information Alert has been posted and will be updated, along with this log, if/when the status of this outage changes. We apologize for the major inconvenience this places on the campus and assure you that we are working as quickly as possible to resolve the problem.

01/20/04 11:10am Replacement part arrived. About an hour of work will be required to prepare it for installation. We expect to install it between noon and 1:00pm. About a 15-20 minute outage in the affected buildings will be required. A 10 minute “warning” will be sent as a Novell Popup Message that will appear on all Windows computers logged into Novell.

01/19/04 10:00am Successful work-around found for the Science Center, Martin House, CTP, Brunk House, Weaver House. All network connectivity to these buildings has been restored as of 10:00am.

01/19/04 08:00am Successful work-around found for the Library. All network connectivity to the Library has been restored as of 8:00am.

01/18/04 03:45pm Cicso confirms via email that the replacement 3508 switch has been prepared for shipment to EMU. It is likely that it will arrive Tue morning, 1/20.

01/17/04 04:00pm TAC opened with Cisco. They will send replacement hardware. Will be to FedEx by Monday and to us by Tuesday. Likely by 11am, but contract only requires by 4pm.

01/17/04 03:00pm Cause found and notice of failure published on Outage Log, Critical Info Alert and Special Notices on Connection page.

01/17/04 02:15pm Problem first investigated by Info Systems

Oracle Calendar Server Database Problems

Calendar Server

There were ongoing problems with the Oracle Calendar Server, particularly for Outlook Connector users. Some users could not see some of their folders (Calendar, Tasks, Contacts, Notes) while using Outlook Connector. There also appeared to be a problem with creating recurring calendar events using Outlook Connector.

It does not seem likely that these problems are related to the Oracle Calendar outage of 1/7/2004.

Oracle provided assistance to run advanced database repair tools and correct the problem. Calendar users are advised to review their data to check for inconsistencies.

Calendar server outage

Calendar Server

8:00am — InfoSys noticed that the calendar server was not responding to connection requests. Diagnosis and repair is ongoing.

10:30am — Info Systems has determined that a major failure within the Linux system occurred, probably sometime before 10pm Tue night. A Network Admin has opened a trouble ticket with Red Hat and several phone conversations with various levels of tech support have occurred. Current level is with a senior systems tech. Because of the nature of this failure we want to be able to determine the root cause before doing any drastic restore to the Linux system software. The Oracle calendar data appears, at this point, to be safe and intact. We have no projected time for the server to be back online.

11:30am — Info Systems confirms that the Oracle Calendar server has been hacked. The system components (other than the oracle application and database) will be restored from the system image backup.

1:00pm — Validation of the Oracle Calendar database fails. It appears that significant corruption of the data has occurred. A restore of the database from the backup of 4:00am, Tues, 1/6/04 will be performed.

2:30pm — Restore of Oracle Calendar database is complete and appears to be valid. Jeremy Good will send an email to all current Oracle Calendar users with details of what has occurred and the state of their data as of now. Any items entered or updated in Oracle Calendar after 4:00am Tues, 1/06, have been lost. It is possible that persons who have PDA devices that they synchronized after 4:00am Tues may be able to load these items from the PDA back into the Oracle Calendar. However, it will be necessary to contact Jeremy to confirm the proper procedure to use to do this.

NOTE: Although the Oracle Calendar server has been returned to service Info Systems must now load a Red Hat Linux kernel patch to fix the vulnerability that was exploited by the hacker who caused this damage. This process will be done after 5:00pm today (Wed, 1/7). Until that is done all access to the Oracle Calendar server outside the EMU firewall will be closed and will only be opened after confirming the system is properly functioning on Thu morning, 1/8. An entry to that effect will be added to this outage log on Thu morning.

POSTPONED: ReConfigure Internet Connection for Increased Capacity

Internet

POSTPONED: The following procedure, originally scheduled for 12/16/03, will be postponed until January due to unavailability of services from our Internet supplier until then.

J.Rutt — 12/12/03
———————–
The EMU Internet connection will be switched from the current configuration of two DS1 lines that provide 3 megabits per second (mbps) capacity to a single ethernet connection of 4mbps. This new connection provides higher capacity (can be increased up to 10mbps) for a lower cost per megabit.

This change has been prompted by the project to implement Video TeleConferencing (VTC) which is being made possible by the LEAP Lilly grant for the seminary.

Network administrators will be working with consultants from STG of Winchester to make this conversion. It is estimated that about 1 hour of actual down-time will be required.

During the outage all access to the Internet will be disrupted. The email servers will remain operational but all incoming and outgoing mail will be held in various queues.

Persons outside EMU will NOT be able to get to the EMU web servers (EMU Home page, Blackboard, WebMail, WebCalendar).