Monthly Archives: June 2004

Outbound Long Distance and Inbound 8XX Outage

6/28 – In order to switch long distance carriers (From ITC/Deltacom to Qwest) the PBX that controls our phone system needs to be reconfigured. The duration of the outage should be short, hopefully less than 20 minutes, if we do not have any complications with the T1 that connects our system to Qwest’s.

As of 1:30 P.M the switch over is on hold due to problems with the local loop, further action will not be taken until Qwest contacts us. All systems are currenly functioning properly with BTI services.

4:00 P.M, June 28 – Qwest has opened up a trouble ticket with Verizon to address the local loop issue. We will not switch service today.

Switchover has been rescheduled to start Wednesday, June 30 at 9:00 A.M.

9/30 – Problems with our equipment, we are waiting to here back from our vendor for additional support. We are still utilizing service from BTi. EXPECT PERIODIC OUTAGES AS WE MAKE CONFIGURATION CHANGES.

RESCHEDULED FOR FRIDAY, JULY 2 @ 9:00 A.M

AS OF 11:00 A.M.- All outbound long distance is now working on the Qwest network.

FS rebooted

A couple users reported that server FS was unreachable. Greg was unable to ping the server, and attempting to ping the router from the server command line caused the console to become unresponsive.

I had Greg, via phone, shut down NovaNet and force a “down” command from the emergency console menu. After at least five minutes the server still was not down. Greg cycled the power, and the server started and appeared to be operating properly.

Novell 6.5 Server Upgrade (ST) – Students

This upgrade went amazingly well. The Student Server (ST) was returned to service about 5:30pm, Thu, 6/24.

============================================
Major work to upgrade the Novell servers to version 6.5 will be performed starting Fri, June 18. We have developed a plan that will limit the number of hours of downtime and the number of users affected at any one time. Specific workdays include:

All-Day Fri, June 18: Faculty/Staff Server (FS)

All-Day Thu, June 24: Student Server (ST)

Sometime the week of 6/28: Special Applications (FSAPPS) for Goldmine, TMA, Tick-it, Preservation Plus (Info Systems will post the planned date by end of day Friday, 6/25)

IMPORTANT: “All-Day” means the outage will likely begin 6-7am in the morning and last until 5-6pm. It also means that in order to complete an undisturbed backup of the systems users should NOT be using the systems after 9pm of the preceding day.

——————————-
Details for the Thursday, June 24 Outage:

USERS AFFECTED:
– Students

SYSTEMS AFFECTED:
– Student Network disk drives (P:, G:, Z:, etc), Network Printing
– Faculty Network disk drives on Student (ST) server (V:, W:)
– Computer labs for Student logins

SYSTEMS THAT SHOULD REMAIN FUNCTIONAL:
– Internet Access
– EMU Website (www.emu.edu)
– Email
– Oracle Calendar
– Student Info System (AS400, Jenzabar)
– Blackboard
– Library catalog (Sirsi)
– Faculty network drives (P:, Z:, etc)
– Faculty/Staff network drives other than those listed in “SYSTEMS AFFECTED” above.

COMMUNICATIONS: During the outages we will be posting status reports to the Critical Info Alert system which will display on the EMU Connection page. The voicemail greeting on the EMU Helpdesk (x4357) will also reflect the current system status.

ONE MORE CAVEAT: This is a very complicated upgrade. Info Systems staff have been testing and planning for it since early March. At this point we believe the schedule outlined above is reasonable and practical for us to achieve. However, it is possible that we will encounter last minute difficulties that will require us to adjust these dates. It is our strong desire to keep these dates, but there is always risk that a significant problem will develop that would change these plans.

============================================

The Upgrade is completed for ST

Sequence of events that occured:

log on to both testupd and st using the migration wizard.

Several people were still logged in – some had open files. These open files were not backed up – oh well.

Disable the accounts of these users so that the migration of the data would not fail.

Migrated the data

performed the rest of the migration utility.

Installed the rest of the nw 6.5 functions from the “products” CD. Seemed to work, but the “closing programs” dialog box stayed open. Manually closed that windows and rebooted server – seemed to not have any problems.

Web server failed to load. Exported server host certificates and imported them back into LDAP. That seemed to take care of the problem.

During migration of data several users failed – 0k file sizes – due to out of space errors – probably due to compressed file. These users Home directories were restored from tape.

Sweep was not able to be copied from the old server’s directory. I had to go into NDS, delete the ST sophos object, and do a re-install on the server to get it happy.

FTP, ldap, powerchute novastor were all loaded and appeared to work.

Upon reboot of the server, serveral items didn’t load properly. It seems to be due to other items that were loading not being completed (the server is too fast). Changed load order – that seemed to help. Changed load order again, using the “?” to pause the loading of several items. Also changed the value of the “?” to be 15 seconds. Things now seem to load properly.

It appear that if Novastor is loaded, you need to unload it before downing the server to allow the server to down properly.

Novell 6.5 Server Upgrade (FS contingency) [CANCELLED]

See description for outage of June 18, 2004.
————————————–
UPDATE: Fri, June 18, 23:30

The upgrade performed on Fri, 6/18, was successfully completed by 11:30pm. There will be no planned outage for Saturday, 6/19.

Network printers will need to be powered off and back on again in order to properly communicate with the newly installed Faculty/Staff Novell Server (FS).

If users encounter problems with the Novell system on Saturday, 6/19, they should call the Info Systems HelpDesk (x4357 on campus, 540-432-4478 from off-campus) and leave a message. If the problem is deemed to be an “emergency” (i.e. justifies a technician being brought to campus requiring a 30 minute drive) please call the Info Systems emergency line (x4541 from on-campus, 540-432-4541 from off-campus).

Novell 6.5 Server Upgrade (FS) – Faculty/Staff

Upgrade SUCCESSFUL as of 11:30pm, Fri, 6/18
————————————

The major work to upgrade the Novell servers to version 6.5 will be done over a period of a week (Fri, June 18 through Thu, June 24). We have developed a plan that will limit the number of hours of downtime and the number of users affected at any one time. Four specific workdays are planned:

All-Day Fri, June 18: Faculty/Staff Server (FS)
All-Day Sat, June 19: A “Contingency Day” for the Faculty/Staff server

All-Day Tue, June 22: Special Applications (FSAPPS) for Goldmine, TMA, Tick-it, Preservation Plus

All-Day Thu, June 24: Student Server (ST)

IMPORTANT: “All-Day” means the outage will likely begin 6-7am in the morning and last until 5-6pm. It also means that in order to complete an undisturbed backup of the systems users should NOT be using the systems after 9pm of the preceding day.

——————————-
Details for the Friday/Saturday, June 18-19 Outage:

USERS AFFECTED:
– Faculty/Staff

SYSTEMS AFFECTED:
– All Faculty/Staff Network disk drives (P:, G:, X:, Z:, etc), Network Printing

SYSTEMS THAT MAY EXPERIENCE VERY BRIEF (up to 15 min) DISRUPTIONS DURING THE DAY:
– Email access
– Oracle Calendar access

SYSTEMS THAT SHOULD REMAIN FUNCTIONAL:
– Internet Access
– EMU Website (www.emu.edu)
– Student Info System (AS400, Jenzabar)
– Blackboard
– Library catalog (Sirsi)
– Computer labs for Student logins
– Student network drives (P:, Z:, etc)

NOTE: We expect to complete the upgrade to the faculty/staff Novell (FS) server on Friday. However, it is the server that will present us with the most significant challenge and we have purposely selected Friday as its upgrade day so that if problems develop we will have Saturday available to solve them.

COMMUNICATIONS: During the outages we will be posting status reports to the Critical Info Alert system which will display on the EMU Connection page. The voicemail greeting on the EMU Helpdesk (x4357) will also reflect the current system status.

ONE MORE CAVEAT: This is a very complicated upgrade. Info Systems staff have been testing and planning for it since early March. At this point we believe the schedule outlined above is reasonable and practical for us to achieve. However, it is possible that we will encounter last minute difficulties that will require us to adjust these dates. It is our strong desire to keep these dates, but there is always risk that a significant problem will develop that would change these plans.

———————————————-

The Upgrade is completed for FS

The following still need to be considered/checked out:

Whoever did the testing of NFAP – could you check that?

Verify that all is happy with Sophos and FS? I was not able to just copy the sweep directory to get it to work. I had to go into NDS, delete the FS sophos object, and do a re-install on the server to get it happy. The sophos novell install is located on ST vol1.

Right before I left on Friday I noticed that FS didn’t want to down properly. If Novastor wasn’t loaded, then it went down properly. I don’t know if we will have time to try and recreate this on a test server today or if we will just find out if the same thing happens on ST. How I got it to go down was to bring up another console command session and select the option for down server. When it came back up again on one of the logging screens it did say that something didn’t down properly, but everything seemed to be working – so – who knows.

Look into the “power failed” and “power restored” messages from powerchute. For now, I’ll remove the emailing “feature” to get rid of those annoying emails.

Sequence of events that occured:

log on to both testupd and fs using the migration wizard.

Several people were still logged in – some had open files. These open files were not backed up – oh well.

Disable the accounts of these users so that the migration of the data would not fail.

Migrated the data

performed the rest of the migration utility.

had problems logging into the “sys” volume – redid backlinks and the last section of the migration – worked.

installed the rest of the nw 6.5 functions from the OS cd – failed partway through. Had to manually reinstall all the drivers and get it working again.

tried to get back in to the migration wizard – couldn’t – said I needed licenses on FS – but to install the licenses I needed to install iManager – but I needed migration wizard to do that – oh boy.

Tried to get iManager re-installed. Using the products CD – it was reinstalled. Got farther into the screens. Found several problems – a missing line from the apache config file, and had to go through the Nsure configuration utility for the iManger – then it worked.

restart server – was able to get on right away before others autoconnected to get an initial license.
Couldn’t mount any volumes – ran dsrepair – then we could. Thought problem to be with DS or ??. AFter the server was brought up – if DS was bounced – could mount the volumes. This ended up being a license issue – which apparently failed to install properly. After licenses were installed via iManager – all worked fine for volume mounting.

Noticed that all trustees were not there again – ran backlinks and reran last section of migration wizard – this time the process took about 45 min. Completed properly and all was well.

During migration of data several users failed – 0k file sizes – due to out of space errors – probably due to compressed file. These users Home directories were restored from tape.

Sweep was not able to be copied from the old server’s directory. I had to go into NDS, delete the FS sophos object, and do a re-install on the server to get it happy.

FTP, ldap, powerchute novastor were all loaded and appeared to work.