The faculty/staff Novell servers need to be rebooted to fix a few things in preparation for the next upgrade.
FS was rebooted, when it was coming back up it did a hard boot when I tried to change off of the gui screen. It did come back up and ran dsrepair.
FSAPPS was rebooted. When it came back up volume APPS didn’t load w/o doing a vrepair. Then DSREPAIR still would not run. I tried unloading DS and the server crashed. After powered cycling it came back up fine and dsrepair did run.
Blackboard is not responding. The server was shut down and restarted.
Unexpected internet outage, outside of campus.
Due to internet provider
Routine maintenance was performed, as well as gathering of diagnostic information for Oracle support.
NOTE: Lacking a better description, for OUR purposes we are calling this virus episode the EMU SATURDAY VIRUS. It is a member of or variant in the Agobot family of viruses.
Residence Hall wall jacks for the students listed below have been disconnected because it is strongly suspected these computers have a virus that is flooding the EMU network and firewall with thousands of invalid data packets.
These computers need to be fixed before the room connections for them are turned on. students need to get the CD and instructions from the HelpDesk in the Campus Center. Instructions on a sheet of paper that accompanies the CD.
By carefully following the instructions for using the CD, the computer will be fixed and the student must then call the HelpDesk to have the network connection re-enabled. If, for some reason, the student computer again begins to flood the network Info Systems will disable the connection and notify the student to bring the computer to the HelpDesk for further work.
As the list below is updated the reconnected computers will be removed from this list. Computers shown on this list are disconnected as of the update time at the top of the list.
Students whose computers remain disconnected as of Tuesday, 1/29, 08:45
———————————————————————–
———————————————————————–
STUDENTS: IF YOUR NAME AND/OR WALL JACK IS ON THE LIST ABOVE — DO NOT ATTEMPT TO MOVE YOUR COMPUTER TO A DIFFERENT WALL JACK. YOUR COMPUTER HAS BEEN IDENTIFIED AS ONE THAT IS TAKING DOWN THE NETWORK! IF YOU HAVE ALREADY MOVED IT — PLEASE UNPLUG IT IMMEDIATELY!!
Problems similar to those that began around 11:00pm, Fri 1/23, seem to have returned.
Info Systems brought a network administrator to campus at noon and began debugging the problem. At about 1:15pm it was determined that the problem is caused by a number of computers in the Residence Halls flooding the network with data that is recognized by the firewall to be “problem connections”. The firewall then started to deny these connections but the volume is too great for it to deal with, its memory filled up and inconsistent network behavior developed in many areas.
At 1:15pm the network administrators shutdown all network connectivity going into and out of the entire Residence Hall network segment. We will try to isolate the computers that are causing the problem to see if we can restore network connectivity for the rest of the Residence Hall connections. This may take several hours.
A number of odd characteristics have been observed on the network during the past week, beginning with the extreme slowdown last Monday evening (1/19) that lasted about 4 hours. Another observation that Info Systems made on Thursday and Friday was that a number of computers in the Residence Hall network segment were transmitting packets of unusual data that looked somewhat like a virus. One of these computers was brought into the HelpDesk area and examined by technicians to see if the virus could be identified. A scan of that system revealed no identifiable virus but an unusual fileset was found. However, tech support did submit a notification to Sophos, asking them to comment on the findings. As of 1:00pm today (1/24) no response has been received from Sophos.
As of 2:00pm (1/24) we believe that there are a number of computers on the Residence Hall LAN segment that have something on them that is sending lots of “bad data” to the network that is choking the firewall.
By 4:00pm the problem computers were identified and their connections to the network were disabled. Info Systems will work with these students on Mon, 1/26, to find a solution to clean their computers and re-enable their network connections.
About 9:15am, Sat, 1/24, Info Systems became aware that the EMU Internet connection was very slow, or non-existent. There were other anomalies such as problems accessing DNS servers, web server and ftp servers. Ping commands worked for some hosts and not others. No web pages could be accessed from off campus and accessing them from on-campus computers was very slow.
Diagnostic procedures lead to a possible problem with the EMU firewall. The device was rebooted about 10:10am and all network services appeared to begin to function normally.
A new type of Internet connection will become available to EMU that will provide more flexibility in meeting future Internet connectivity needs. This new service will require some downtime for the Internet connection to EMU.
An outside consultant will assist with this operation. Coordination will be required with our current Internet provider, nTelos.
We anticipate that about an hour of interuption will be required, however, there may be lingering affects that will require up to several hours until the up-stream DNS servers learn of this new type of connection. However, no email message should be lost, only delayed.
Systems Affected include: Internet connection, email, off-campus access to web systems on-campus (i.e. Blackboard, Campus Web, WebMail). All on-campus access to these web systems will NOT be affected.
About 4:35pm today (Mon, 1/19) EMU began experiencing slow Internet response times. The situation continues to worsen.
Info Systems is aware of the problem and is attempting to identify the cause. There is no projected time for resolution of the problem.
At the present time you may or may not be able to access off-campus web pages.
————————-
Investigation revealed that there was an unusually high number of TCP packets being both sent to and being received from the Internet connection (i.e. nearly 3X normal rate both in and out). This info was obtained from discussions with an ISP engineer. Total bandwidth, however, was within reasonable range. The only way to debug further was by use of advanced diagnostic equipment which was not easily available at the time.
About 9pm performance returned to near normal conditions without explanation.
The Student Information System (iSeries, Campus Web) will be unavailable while routine PTFs (Program Temporary Fixes) are applied to the system. A two to three hour interuption is anticipated.
The systems affected will be Jenzabar application software (TEAMS2000) and CampusWeb (https://campusweb.emu.edu). CampusWeb provides student access to their financial information (billings and FinAid) as well as final class grades.
No other systems are affected by this outage.