All posts by admin

Problems with Long Distance provider

Phone System

Long distance outage from 3:00-4:30, rerouting changes were made. Users can now make long distance calls, but inbound long distance service is not working.

Our service provider has been contacted. We hope to have the problem solved soon.

9:00 A.M 8/20/04 – The circuit appears to be stable. Long distance service is currently working. We will monitor the traffic until 5:00 P.M. this afternoon. If no problems arise we will close our trouble ticket with Qwest at that time.

Webmail outage

Email

10:30AM, MON 3/22: A small routine upgrade procedure failed on the webmail server this morning. While this was being fixed, an additional problem occurred which is very serious and will take a few hours to correct. This affects only accessibility and will not result in lost email.

04:00PM, MON 3/22: Indications are that the WebMail server hardware failed during the routine procedure this morning. WebMail services will be re-deployed to the mail server itself. One server will temporarily be doing the work of two. Performance will be impacted — the extent of the slowed performance is unclear.

06:00PM, MON 3/22: Webmail services are available for on-campus users, but off-campus users will not be able to access webmail until Tuesday, 3/23. The temporary solution will result in slower e-mail server performance. Please be patient. One server is now doing the work previously done by two. Info Systems is working to develop a plan for returning a second e-mail server to service with appropriate hardware.

11:00AM, TUE 3/23: Webmail is available both ON and OFF campus. However, performance is slow and SENDING e-mail messages may fail in some instances. Technicians are working to reconfigure server hardware to address these performance issues. It is anticipated that this work will be completed by end of day today (Tue, 3/23).

02:30PM, TUE 3/23: Webmail services moved off the main mail server to another web server. Main mail server should be able to reduce the queues of messages for delivery. Quirks with WebMail (i.e. attachments, groups, inbox display) should be fixed. Users should reboot their computers before connecting to WebMail to assure proper connectivity.

Novell Server Service Pack 7 Upgrade

Network

Novell Service Pack 7 needs to be installed.

The planned outage for this Service Pack upgrade is Saturday morning, 03/20/2004.

The process will require several hours of down time. Systems affected will be Novell file servers and all other systems that authenticate with LDAP (i.e. email, calendar, certain web services).

Plans are to begin shortly before 7am on Saturday, 3/20. If all goes well all systems should return to service by 1:00pm.

During the outage the following network services will NOT be available:

NOVELL FILE/PRINT: (P:, G:, Z: drives and network printers)

EMAIL: LDAP authentication will be down so users cannot login to email. No email will be lost because it will be held in queue.

CALENDAR: The Oracle Calendar LDAP authentication will be down. Users will not be able to login to the calendar server.

COMPUTER LABS: Windows computers located in computer labs will not be operational because they require LOGIN to operate. LOGIN is provided by Novell which will not be available during the outage.

CERTAIN WEB SERVICES: A few functions on the EMU web site require user login (i.e. access to web directory from off-campus). These rely on LDAP which will be unavailable.

All other network services should remain operational (i.e. Internet access for browsing and IM).

_______________________________________

All systems were updated – no major problems.

Both NS and ST had to be powered off during restart. Is this due to the KVM switch again?

Ran DSrepair on FS – no problems

Changed the time server for ST – the existing one wasn’t responding.

Upgraded FS first, then FSAPPS and LD (at the same time), then ST and NS (at the same time).

On ST in the NWSERVER directory, renamed DELLMGR.NLM to DELLMGR.NL1.

Since the memory hadn’t arrived for LD, it wasn’t added – will need to be added some morning after we ever get it. This will max the server out to 2G.

The “backup” UPS wasn’t added to the second rack since the power source wasn’t able to be determined – waiting on phys plant to either verify what else is on that breaker or to add a new 20A circuit.

Novell eDirectory Security Patch Install – LDAP/Novell Down

File Server

A major security patch must be applied to the Novell Directory Services (NDS) system. The process will require several hours of down time. Systems affected will be Novell file servers and all other systems that authenticate with LDAP (i.e. email, calendar, certain web services).

Plans are to begin shortly before 7am on Saturday, 2/28. If all goes well all systems should return to service by 2:00pm.

During the outage the following network services will NOT be available:

NOVELL FILE/PRINT: (P:, G:, Z: drives and network printers)
EMAIL: LDAP authentication will be down so users cannot login to email. No email will be lost because it will be held in queue.
CALENDAR: The Oracle Calendar LDAP authentication will be down. Users will not be able to login to the calenar server.
CERTAIN WEB SERVICES: A few functions on the EMU web site require user login (i.e. access to web directory from off-campus). These rely on LDAP which will be unavailable.

Concurrent with this upgrade technicians will be upgrading Linux servers. Each server will be down for a short time and the entire upgrade process is expected to take less than two hours–between 8:00am and 10:00am.

All other network services should remain operational (i.e. Internet access for browsing and IM).

During the EDirectory upgrade, several items were noted – they are as follows:

On server FS:
Several schema changed produced an error – they were SASSecretStore:Key, SASSecretStore:Data, PKIStore:keys, NDSPKI:Keystore. On doing a check with Novell’s knowledgebase these could be errors in the install of 8.7.1 where the attributes should be marked as hidden, but are not. They state that this error can be ignored and processing continue (TID 1008666) and that Novell can dial-in to correct this problem.
There was also a problem wher “PKI Install encountered an error -641” and “NMAS object could not be installed – error -641”. After the installation I checked and things seemed to be installed correctly – I’m hoping these errors were because of the “hidden field” problem stated earlier.

On server LD:
A schema error occured with houseIdentifier. Again – according to Novell’s knowledgebase, this error is because the shema wasn’t updated correctly with the schema enhancements of DSREPAIR. We know this not to be the case, so after everything was completed on all servers I issued a command for LD to abtain a new schema from the tree. There were also errors where, during the upgrade serveral files were not replaced because newer ones were already on the server. These files are as follows: JCERT.JAR, JNET.JAR, JSSE.JAR

On server FSAPPS:
“An error occured while installing product LDAP – error -254”. This occured during the eDirectory upgrade. The server also crashed while it was trying to reboot (during the upgrade). This required me to powercycle the machine to continue.

On server ST:
The server hung after the reboot – it didn’t show a console screen. I was able to spawn another console process and dismount all the volumes and power it off.

After all servers were upgraded, I ran DSREPAIR to check on timesync and synchronization errors – ST wouldn’t sync – error 625. After running an unattended repair it seemed to work fine.

Also noted were SOMETIMES when DSREPAIR was run (unattended full) an error would occur on the console screen that stated:

NLSLSP: main was unsuccessful
SERVER-5.00-205: Module NLS FLAIM Database Engine cannot be unloaded at this tim
e.
Module NLSLSP.NLM is being referenced
You must unload NLSFLAIM.NLM before you can unload NLSLSP.NLM
You must unload NLSTRAP.NLM before you can unload NLSLSP.NLM

2-28-2004 12:04:45 pm: SERVER-5.0-1400
Error unloading killed loadable module

This occured on most servers and didn’t occur all the time. I couldn’t find anything regarding this from Novell and all seemed to still be fine on the server afterwards – so – time will tell if we need to deal with this.

Short Novell Server Outage – faculty staff servers

File Server

The faculty/staff Novell servers need to be rebooted to fix a few things in preparation for the next upgrade.

FS was rebooted, when it was coming back up it did a hard boot when I tried to change off of the gui screen. It did come back up and ran dsrepair.

FSAPPS was rebooted. When it came back up volume APPS didn’t load w/o doing a vrepair. Then DSREPAIR still would not run. I tried unloading DS and the server crashed. After powered cycling it came back up fine and dsrepair did run.