Routine maintenance was performed, as well as gathering of diagnostic information for Oracle support.
Monthly Archives: January 2004
STUDENT COMPUTERS DISCONNECTED TO SAVE THE NETWORK!
NOTE: Lacking a better description, for OUR purposes we are calling this virus episode the EMU SATURDAY VIRUS. It is a member of or variant in the Agobot family of viruses.
Residence Hall wall jacks for the students listed below have been disconnected because it is strongly suspected these computers have a virus that is flooding the EMU network and firewall with thousands of invalid data packets.
These computers need to be fixed before the room connections for them are turned on. students need to get the CD and instructions from the HelpDesk in the Campus Center. Instructions on a sheet of paper that accompanies the CD.
By carefully following the instructions for using the CD, the computer will be fixed and the student must then call the HelpDesk to have the network connection re-enabled. If, for some reason, the student computer again begins to flood the network Info Systems will disable the connection and notify the student to bring the computer to the HelpDesk for further work.
As the list below is updated the reconnected computers will be removed from this list. Computers shown on this list are disconnected as of the update time at the top of the list.
Students whose computers remain disconnected as of Tuesday, 1/29, 08:45
———————————————————————–
———————————————————————–
STUDENTS: IF YOUR NAME AND/OR WALL JACK IS ON THE LIST ABOVE — DO NOT ATTEMPT TO MOVE YOUR COMPUTER TO A DIFFERENT WALL JACK. YOUR COMPUTER HAS BEEN IDENTIFIED AS ONE THAT IS TAKING DOWN THE NETWORK! IF YOU HAVE ALREADY MOVED IT — PLEASE UNPLUG IT IMMEDIATELY!!
Problems with Internet Connection – Again at 11:00am, Sat, 1/24
Problems similar to those that began around 11:00pm, Fri 1/23, seem to have returned.
Info Systems brought a network administrator to campus at noon and began debugging the problem. At about 1:15pm it was determined that the problem is caused by a number of computers in the Residence Halls flooding the network with data that is recognized by the firewall to be “problem connections”. The firewall then started to deny these connections but the volume is too great for it to deal with, its memory filled up and inconsistent network behavior developed in many areas.
At 1:15pm the network administrators shutdown all network connectivity going into and out of the entire Residence Hall network segment. We will try to isolate the computers that are causing the problem to see if we can restore network connectivity for the rest of the Residence Hall connections. This may take several hours.
A number of odd characteristics have been observed on the network during the past week, beginning with the extreme slowdown last Monday evening (1/19) that lasted about 4 hours. Another observation that Info Systems made on Thursday and Friday was that a number of computers in the Residence Hall network segment were transmitting packets of unusual data that looked somewhat like a virus. One of these computers was brought into the HelpDesk area and examined by technicians to see if the virus could be identified. A scan of that system revealed no identifiable virus but an unusual fileset was found. However, tech support did submit a notification to Sophos, asking them to comment on the findings. As of 1:00pm today (1/24) no response has been received from Sophos.
As of 2:00pm (1/24) we believe that there are a number of computers on the Residence Hall LAN segment that have something on them that is sending lots of “bad data” to the network that is choking the firewall.
By 4:00pm the problem computers were identified and their connections to the network were disabled. Info Systems will work with these students on Mon, 1/26, to find a solution to clean their computers and re-enable their network connections.
Extreme Network Slowdown for 1 to 2 Hours, Sat, 1/24
About 9:15am, Sat, 1/24, Info Systems became aware that the EMU Internet connection was very slow, or non-existent. There were other anomalies such as problems accessing DNS servers, web server and ftp servers. Ping commands worked for some hosts and not others. No web pages could be accessed from off campus and accessing them from on-campus computers was very slow.
Diagnostic procedures lead to a possible problem with the EMU firewall. The device was rebooted about 10:10am and all network services appeared to begin to function normally.
Internet Connection Reconfiguration
A new type of Internet connection will become available to EMU that will provide more flexibility in meeting future Internet connectivity needs. This new service will require some downtime for the Internet connection to EMU.
An outside consultant will assist with this operation. Coordination will be required with our current Internet provider, nTelos.
We anticipate that about an hour of interuption will be required, however, there may be lingering affects that will require up to several hours until the up-stream DNS servers learn of this new type of connection. However, no email message should be lost, only delayed.
Systems Affected include: Internet connection, email, off-campus access to web systems on-campus (i.e. Blackboard, Campus Web, WebMail). All on-campus access to these web systems will NOT be affected.
Internet Connection is Slow or Non-Existent
About 4:35pm today (Mon, 1/19) EMU began experiencing slow Internet response times. The situation continues to worsen.
Info Systems is aware of the problem and is attempting to identify the cause. There is no projected time for resolution of the problem.
At the present time you may or may not be able to access off-campus web pages.
————————-
Investigation revealed that there was an unusually high number of TCP packets being both sent to and being received from the Internet connection (i.e. nearly 3X normal rate both in and out). This info was obtained from discussions with an ISP engineer. Total bandwidth, however, was within reasonable range. The only way to debug further was by use of advanced diagnostic equipment which was not easily available at the time.
About 9pm performance returned to near normal conditions without explanation.
Student Info Systems (SIS) PTF Installation (aka AS400)
The Student Information System (iSeries, Campus Web) will be unavailable while routine PTFs (Program Temporary Fixes) are applied to the system. A two to three hour interuption is anticipated.
The systems affected will be Jenzabar application software (TEAMS2000) and CampusWeb (https://campusweb.emu.edu). CampusWeb provides student access to their financial information (billings and FinAid) as well as final class grades.
No other systems are affected by this outage.
MAJOR NETWORK FAILURE – South End of Campus
NOTE #1: Last updated 01/20/04 11:30am The most recent updates are in BOLD text below.
NOTE #2: All available work-arounds have been completed. Marketing Services (Communications Dept) and Development Offices will remain without network connectivity until the permanent fix is installed sometime on Tuesday, 1/20.
A Cisco fiber switch located in the Library failed completely about noon today (1/17). This is one of two major fiber switches on campus, providing network service to all the buildings in the southeast corner of campus.
Info Systems is aware of the problem and is working to obtain a replacement device. It is not clear how long this will take but it is unlikely to be before Tuesday, 1/20.
Network connectivity to the following buildings is affected:
Library
Science Center
Marketing Services (Communications)
Development
Martin House
CTP
Weaver House
Only connectivity to these buildings is affected. All servers remain functional (i.e. Internet, email, calendar, SIS(As400), web, Blackboard, Novell). You just can’t get to them from these buildings.
A Critical Information Alert has been posted and will be updated, along with this log, if/when the status of this outage changes. We apologize for the major inconvenience this places on the campus and assure you that we are working as quickly as possible to resolve the problem.
01/20/04 11:10am Replacement part arrived. About an hour of work will be required to prepare it for installation. We expect to install it between noon and 1:00pm. About a 15-20 minute outage in the affected buildings will be required. A 10 minute “warning” will be sent as a Novell Popup Message that will appear on all Windows computers logged into Novell.
01/19/04 10:00am Successful work-around found for the Science Center, Martin House, CTP, Brunk House, Weaver House. All network connectivity to these buildings has been restored as of 10:00am.
01/19/04 08:00am Successful work-around found for the Library. All network connectivity to the Library has been restored as of 8:00am.
01/18/04 03:45pm Cicso confirms via email that the replacement 3508 switch has been prepared for shipment to EMU. It is likely that it will arrive Tue morning, 1/20.
01/17/04 04:00pm TAC opened with Cisco. They will send replacement hardware. Will be to FedEx by Monday and to us by Tuesday. Likely by 11am, but contract only requires by 4pm.
01/17/04 03:00pm Cause found and notice of failure published on Outage Log, Critical Info Alert and Special Notices on Connection page.
01/17/04 02:15pm Problem first investigated by Info Systems
TMA Software Upgrade
A new version of the TMA software is available and TMA recommends that all customers upgrade. TMA will be unaccessible through the desktop client and web interfaces during this time.
Web and mail server maintenance
The mail server, web server, and several other core servers will be taken off line for system maintenance. Individual servers may be available during this time, but this is not assured and should not be relied upon.