Monthly Archives: October 2003

NS server disk upgrade

The Novell server NS had larger disks installed.

The four 36 gig disks were replaced with four 73 gig disks. This meant the raid needed to be rebuilt. Since the data section was smaller than 100G, I was able to use server magic. Following is the order of what was done:
I created a tape backup of the server
Ccreated an image of the sys, c: partition.
Deleted most of the data from the data partition.
Created an image of the data partition.
Swapped out disks.
Created the new array and logical drives using server magic – which allows me to increase the size of the netware partition.
Installed both images.
Used Novastor to restore the data to the “SERVIMG” and “IMAGES” volumes.
(this created some problems when trying to restore data to more than 1 volume at a time.)

This now doubled the space to store system images.

Dan Marple Jr.

Web Instability

The database server is running again. This impacted the home page, Campus Connection page, and other systems. Updates if any will be posted here later.
——-
Update 11am: The problem began with Apache and memory. It then impacted other servers like the database. Once Apache was restarted it appears fine. We are looking to upgrade the memory in the next weeks.

24 Hour Faculty/Staff Novell Hardware Upgrade Outage

Info Systems proposes to install new hardware for the Faculty/Staff Novell server (FS) on Tuesday, 10/21/2003 during the Fall Semester break. This process will begin with a full quiescent backup (i.e. all users logged out of the server). All users should be logged off the Faculty/Staff Novell server by 6:00pm on Monday evening, October 20.

The installation process will take most of the day on Tuesday, 10/21. Users should NOT expect to be able to access the FS network disk space (i.e. faculty/staff P: G: Z: drives) during normal working hours on Tuesday, 10/21.

While the actual hardware upgrade involves only the FS server, a precautionary full Novell Systems backup will be performed between 6:00am and 9:00am on Tuesday, 10/21. During this time ALL Novell servers, will be off-line which means that the EMU email and calendar systems will be unavailable during this time (i.e. email and calendar access authorization services are provided via LDAP through the Novell NDS service).

– – – – – – – – – – – – – SUMMARY – – – – – – – – – – – – –

F/S Novell Server
UNAVAILABLE 6:00pm Mon, 10/20 thru 11:30pm Tue, 10/21

ALL Novell Servers
UNAVAILABLE 6:00am thru 12:00pm Tue, 10/21

EMail and Oracle Calendar Servers
UNAVAILABLE 6:00am thru 12:00pm Tue, 10/21

– – – – – – – – – – – – – – – – – – – – – – – – – – – – – –

IMPORTANT NOTE: A more detailed description of the outage timetable is available on a web page here. You are urged to view click the link to see the detailed information.

===========================================================
Outage Follow Up Summary:

Delays were encountered from the start on Tues morning (10/21) due to problems with the overnight backup procedures plus an unexplained crash of the FSAPPS server. By noon the FS migration process was at least 3 hours behind schedule.

Migration process was started about 12:30pm and finished about 6:30pm. While doing various validation and maintenace procedures a significant error was discovered about 7:30pm (-608, Volume ID number conflict). Research was done and a few procedures attempted without success. Jeremy Good was called and joined the effort about 9:00pm. This lead eventually to placing a support call to Novell (available only because Jeremy has a CNE designation). By 10:15pm a Novell engineer was available and soon identified the problem — a known flaw in the architecture of the Migration Wizard. He provided a procedure which successfully corrected the problem.

By 11:30pm the system was stable but it was the end of a very long day, particularly for Dan Maple Jr. Sophos and backup software were then installed and a server image was done along. The final procedure was to launch a one-time full backup to tape.

One user item needed to be recreated (Brenda Fairweather) because her user record was purposely deleted as part of the testing procedure in the effort to find a fix to the -608 error. Voicemail was left with her alerting her to the need to use our assigned password and then go change it to one of her liking.

— Management summary by Jack Rutt —

FS hardware migration outage summary

Outage Follow Up Summary:

The backup for FS didn’t occur as expected. Seems Novastor virtual expects weekly tapes if the job is defined as a weekly no mater what day the job occurs on. Got FSAPPS running again (crashed for some reason), and finally got a full DSREPAIR to work on it. Downed all servers but NS, imaged the sys partitions to NS. Downed NS and imaged it to FSAPPS.

Brought all servers back online, started a “real” tape backup of FS. After it completed, started the migration process. The migration process seemed to go relatively smooth. After it completed around 6:30, post migration procedures were started. DSREPAIR failed with a 608 error when “checking volume objects and trustees”. This occurred because of an objectID conflict between the volume and another object. Of course this was the case for ALL the volumes. 1 was in conflict with a container, the other were in conflict with users. After various procedures and research on Novells site failed to correct the problem, Jeremy Good came in and we contacted Novell. Trevor stated the “error” was known and the best way to resolve the problem was by deleting the user objects and recreating. (yea right). This was tried with one user – it worked. He then found a utility to change the objectid of a volume. “voleid.nlm” This was tried and worked on the other volumes. Using this utility, we changed the volume object id’s 1 at a time then ran dsrepair (check volume id’s and trustees), then repeated the cycle for the rest of the volumes. It was stated that this NLM doesn’t appear to work for NSS volumes – luckily we have traditional volumes. This process was completed around 11:30PM.

Novastor, pwrchute, and sophos were added to the FS server and appeared to run fine. Changes were made to the autoexec.ncf to allow for automatic loading of these and a few other Novell nlms. An image was made to FSAPPS and a full backup was started to “real” tape. What had yet to be done was go through the “Performance tuning” tid, and check a few other parameters.

— technical summary by Dan Marple Jr. —

AS400 Hardware Upgrade: Over-Night Outage, Wednesday, 10/15

A complete hardware upgrade for the AS400 will be installed over-night, Wednesday, 10/15-16. The AS400 will be taken off-line at 5:00pm on Oct 15. If all goes well the system should be operational by 8:00am on Thursday, Oct 16. During the outage all access to the AS400 will be unavailable.

The new hardware will be an IBM iSeries 810-2465 server. IBM no longer markets an AS400 computer. It has been replaced with the iSeries which has been optimized for on-line web connectivity.

The performance increase from this new hardware will be substantial. It should meet the needs of EMU for at least the next 3 years.

NOVELL SERVERS: 10 Minute Outage for REBOOT, 5:30pm, Today

A problem has been identified following the Novell software upgrade of Saturday that will now require a reboot of all the Novell servers.

Info Systems will reboot the Novell servers at 5:30pm today. A message to save any open documents and logoff Novell will be broadcast 10 minutes before the servers go down. The outage will last about 10 minutes.

This only affects network disks, network printers and authentication to email and calendar.

If users do NOT logoff of Novell by 5:30pm it would be best to reboot their computers in order to properly reconnect to the Novell servers after the outage.