For The Love Of UNIX

2009-09-17T13:52:00.005-04:00

SENDING WINDOWS EVENTS TO A SYSLOG SERVER

Many of us know and love (or hate) syslog and syslog-ng for UNIX based operating systems. The ability to collect and monitor all UNIX based system logs in a centralized repository has been a standard feature of UNIX systems for ages now. This is great, if all you have are UNIX or Linux systems.

After much trial and error, I have found what I believe to be the simplest and most cost-effective method of deploying centralized logging for Windows systems. I have documented everything here as best as I can remember.

You will need the following:

Windows systems that you want to get log data from
A syslog server to send the data to
The "Eventlog to Syslog" utility

If you already have a syslog server, and are just here to get your Windows server reporting to it, then proceed to the EVENTLOG TO SYSLOG UTILITY section.

If you simply want the deployment script, check out the DEPLOYMENT SCRIPT section.

If you don't have a syslog server yet, you can follow my directions in the SYSLOG SERVER section or use any of the instructions out there on the web.

If you are just here to configure swatch, check out the SWATCH CONFIGURATION section.

If you're here for the whole lot, then read on from this point.

EVENTLOG TO SYSLOG UTILITY

The good folks at Purdue University have written and fully documented a wonderfully simple app called, "Eventlog to Syslog". They have written it for both 32-bit and 64-bit Windows systems, and even provided the source code. Oh, and it's free.

You can find the project page for this app here: https://engineering.purdue.edu/ECN/Resources/Documents/UNIX/evtsys/

I recommend downloading all available files for posterity- You never know if and when they will stop providing access to this utility. I created a hidden network share that contains all installation files and documentation for this utility, as well as the deployment/update scripts. I will, of course, be sharing these scripts as well.

To manually install the utility:

Uncompress "evtsys.DLL" and "evtsys.exe" to %systemroot%\system32.
At a command prompt, enter "%systemroot\system32\evtsys.exe -i -h hostname", where "hostname" is the IP address or DNS name of your syslog server.
At the command prompt or through the services GUI, start the "Eventlog to Syslog" service.

The service is now running and forwarding all Eventlog entries via UDP port 514 to the hostname you specified. Obviously, since this is UDP there will be no error messages if the remote syslog server is down.

You can also specify custom facilities for Eventlog to Syslog, but I won’t go into them here. Visit the project homepage.

DEPLOYMENT SCRIPT

Whether you have dozens or hundreds of Windows servers to monitor, a scripted deployment is the easiest and fastest way to go. If you have thousands of servers to monitor, you should be taking your time with this and plan thoroughly.

Basically, I have a script that I use for installing the service and starting it remotely. I like to use a tool called "Hyena" to schedule the execution of the script on multiple servers as AT jobs. A similar tool is "Dameware". If you're really clever, you can use WMI scripting or even domain logon scripts.

Whatever method you choose for scheduling and executing the script, the code remains the same.

Prerequisites: You will need to have the “Eventlog to Syslog” executable, DLL, and deployment script all in one file share on your network for this script to work properly. Edit the script and change all paths containing the word “fileserver” to suit your needs. Obviously, this should be the path to the aforementioned network file share.

I should mention that I had initially tried using another free utility called “Winlogd”, which can be found here: http://edoceo.com/creo/winlogd. This utility works okay, but it had some shortcomings. It required .Net, and it would not run on 64-bit systems. It also had some issues on some of my Windows 2003 R2 SP2 systems as well. The biggest problem was that it did not report uptime events from Windows 2003 servers correctly at all. This completely hosed my logs.

So, just in case anyone reading this has tried Winlogd as well, I have included an uninstaller routine in my batch file. It won’t hurt anything to leave it in, but if you really want it gone, just remove the WLDCHECK, WLDREPORT, and WLDREMOVE sections from the batch file.

To summarize the script, it checks to see if “Eventlog to Syslog” already exists on the remote system. If so, it reports this in a log and quits. If not, it checks to see if “Winlogd” is on the remote system. If “Winlogd” is found, it removes it, logs it, and then installs “Eventlog to Syslog”. Finally, it starts the “Eventlog to Syslog” service and logs it. By the way, I should also mention that this script will handle terminal servers with no issue. It is set to change the user mode to “/Install” where appropriate and then set it back to “/Execute” when finished.

So, without further delay, here is my “install.bat” script:

****************************************************************************************

:: Check to see whether we need to continue

IF EXIST %systemroot%\system32\evtsys.exe (GOTO :EVTREPORT) ELSE GOTO :WLDCHECK

:EVTREPORT

echo "Evtsys is already installed on %computername%" >> \\fileserver\syslogd$\report.txt

GOTO :END

:WLDCHECK

IF EXIST %systemroot%\system32\winlogd.exe (GOTO :WLDREPORT) ELSE GOTO :DOIT

:WLDREPORT

echo "Winlogd is installed on %computername%" >> \\fileserver\syslogd$\report.txt

GOTO :WLDREMOVE

:WLDREMOVE

change user /install

net stop winlogd

%systemroot%\system32\winlogd -u

del %systemroot%\system32\winlogd.exe /Q

change user /execute

GOTO :DOIT

:DOIT

change user /install

copy \\fileserver\syslogd$\32-bit\evtsys.* %systemroot%\system32

%systemroot%\system32\evtsys -i -h 10.221.2.24

net start "Eventlog to Syslog"

change user /execute

echo "Successfully installed on %computername%" >> \\fileserver\syslogd$\report.txt

:END

****************************************************************************************

SYSLOG SERVER

You can likely use any syslog-like system that accepts standard syslog-formatted messages from UDP port 514 in conjunction with the “Eventlog to Syslog” utility. I prefer to use syslog-ng on a Linux box because it’s free, stable, and easy to configure. So, obviously my documentation here will only cover syslog-ng on a Linux system.

I chose to use a Fedora system for this deployment initially, because is easily deployed, free, and there is lots of community support. So, it was a great candidate for testing this. Of course, my *ahem* test system did eventually become my production syslog server. I have recently rebuilt this system on an Opensolaris 10 platform. I will post those handy-dandy directions later.

Here are the steps I took to configure syslog-ng on my Fedora Core 6 system:

As ‘root’, run:

yum install syslog-ng

Accept any dependencies, and voila! Syslog-ng is installed. Now to configure it…

Syslog and syslog-ng can place log entries in various logfiles, depending on what rules you have set up. I prefer to have everything dumped to one big log file that is rotated daily. Once I have a week’s worth of logs, I tarball them, bzip2 them, and throw them up on a fileserver for backup to tape. This is important for PCI compliance, for all you retailers out there, by the way.

Here is my syslog-ng.conf file:

****************************************************************************************

# Syslog-ng configuration file.

###############################################################

# First, set some global options.

###############################################################

options {

keep_hostname(yes);

long_hostnames(off);

sync(1);

log_fifo_size(2048);

};

###############################################################

# Set up our log sources (local events and everything from port 514)

###############################################################

source src {

pipe("/proc/kmsg");

unix-stream("/dev/log");

internal();

};

source network {

udp( port(514));

};

###############################################################

# After that set destinations. (..Everything to one file per day.)

###############################################################

destination std {

file("/var/log/hosts/current/$YEAR-$MONTH-$DAY"

owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)

);

};

################################################################

# Set up logging to flat files. (This actually writes the data to files.)

###############################################################

log {

source(network);

destination(std);

};

log {

source(src);

destination(std);

};

****************************************************************************************

After a week of running the syslog server, if I check the contents of the log directory (/var/log/hosts/current), I see the following:

# ls -alh

total 463M

drwxrwxrwx 2 syslog syslog 12K Aug 4 00:00 .

drwxrwxrwx 5 syslog syslog 4.0K Jul 18 2007 ..

-rw------- 1 root root 245M Jul 27 23:59 2008-07-27

-rw------- 1 root root 633M Jul 28 23:59 2008-07-28

-rw------- 1 root root 640M Jul 30 00:00 2008-07-29

-rw------- 1 root root 613M Jul 31 00:00 2008-07-30

-rw------- 1 root root 610M Jul 31 23:59 2008-07-31

-rw------- 1 root root 515M Aug 1 23:59 2008-08-01

-rw------- 1 root root 232M Aug 2 23:59 2008-08-02

You see seven files; each is named with the year-month-day on which it was created. This ensures that no two log files are ever named the same, and it makes historical research very easy. I want to take a moment to point out the size of these files. Notice that each one of the weekday files is over 600MB. I have less than 200 servers reporting to this syslog server. You can imagine if I had upwards of 500 servers. This is why I mentioned earlier that if you have 1000 or more servers, you need to really take your time and plan this deployment. Tossing that much data around your network tends to have a noticeable effect on performance.

Now, being that these files are 100% plain-old text, they compress very nicely. Here is the log rotation script I use with cron:

****************************************************************************************

#!/bin/sh

#################################

### "syslog_archiver" Written by Malachi McCabe

### The purpose of this script is to move the syslog

### logs to an archive location and compress them. It

### then uploads the compressed logs to another server

### for tape archival to meet PCI requirements.

#################################

### Set some variables.

#################################

CTIME=

CTIME=`date +%Y%m%d`

SUBJECT="Syslog Archival Process"

TO=myemailaddress@work.com

#################################

### Compress and move the logs to the archive directory.

#################################

mkdir /tmp/syslog-tmp

mv /var/log/hosts/current/* /tmp/syslog-tmp/

/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true

/etc/init.d/swatch stop

tar -cvjf /var/log/hosts/archive/syslog-$CTIME.bz2 /tmp/syslog-tmp/*

rm -rf /tmp/syslog-tmp

/etc/init.d/swatch start

#################################

### Upload the archive to the remote server for tape backup.

#################################

cd /var/log/hosts/archive

smbclient \\\\YOURFILESERVER\\syslog$ YOURPASSWORD -U syslogd -W YOURDOMAIN -c "prompt off;put syslog-$CTIME.bz2;exit"

#################################

### Notify us that the syslog archive process has run.

#################################

mailx -s "${SUBJECT}" ${TO} <<-EOF

This week's syslog archive "syslog-$CTIME.bz2" has been uploaded successfully for tape backup.

EOF

****************************************************************************************

I do want to point out that I am uploading my compressed log file archives to a Windows server. For interoperability, I used Samba and a domain service account. In a nutshell, this script will move your current log files to a temporary location, restart syslog, compress the log files, upload them to a Windows file server, and finally email you that the process has completed.

My system runs this script every Sunday at midnight, but you can set yours to whatever you like.

SWATCH CONFIGURATION

So, you have a few Windows servers sending their event log entries to a syslog server now, eh? Great! Now you can search through it using grep, awk, etc to find whatever your looking for. Why not take it a step further, though? Why be reactive instead of proactive? Why not use swatch to monitor your new syslog server for specific Windows events? Here’s how:

First, install swatch as ‘root’:

# yum install swatch

Now, let’s configure it. Here’s my swatch configuration script:

****************************************************************************************

######################################################

### Configuration file for Swatch

######################################################

# Ignore the nextorclean service

ignore /nextorclean/

ignore /domain: HIS/

ignore /domain: ASPECTCC/

# Kernel problems or system reboots

#watchfor /panic|halt/

# echo bold

# mail myemailaddress@work.com,subject=System Reboot

# Logon attempted with disabled account

watchfor /Account currently disabled/

mail myemailaddress@work.com,subject=Logon Attempted With Disabled Account

threshold track_by=$4:$17,type=limit,count=1,seconds=20

# Logon attempted with account locked out

watchfor /Account locked out/

mail myemailaddress@work.com,subject=Five Logon Attempts with Locked Out Account

threshold track_by=$4:$17,type=both,count=5,seconds=20

# Unknown user name or bad password

watchfor /Unknown user name or bad password/

mail myemailaddress@work.com,subject=Five Attempts With Unknown Username Or Bad Password

threshold track_by=$4:$20,type=both,count=5,seconds=20

# SQL service monitoring

watchfor /MSSQLSERVER service/

mail myemailaddress@work.com,subject=SQL Service State Changed

# Manual server reboots

watchfor /initiated the restart/

mail myemailaddress@work.com,subject=Forcible Server Reboot

# Citrix Data Store connectivity

#watchfor /IMA_RESULT_DBCONNECT_FAILURE/

# mail myemailaddress@work.com,subject=Unable to Contact CTX Data Store

# Nextor Service Monitoring (Enterprise Store Server)

watchfor /nnrtl001.+nextor.+service entered the stopped state/

mail myemailaddress@work.com,subject=Nextor Service Stoppage On Enterprise Server

# Nextor Service Monitoring (KoP Store Server)

watchfor /retserv1.+nextor.+service entered the stopped state/

mail myemailaddress@work.com,subject=Nextor Service Stoppage On Retail Server

# This alerts on failed su attempts. This can get annoying if you have a lot

# of boxes and users.

watchfor /\'su root\' failed/

echo bold

mail myemailaddress@work.com,subject=Failed SU To Root

# Full filesystems on Sun Boxes

watchfor /file system full/

mail myemailaddress@work.com,subject=File System Full

# Dell Power Supply Issues

watchfor /Voltage sensor detected a failure/

mail myemailaddress@work.com,subject=Dell Server Power Supply Issue

threshold track_by=$4:$8,type=limit,count=1,seconds=60

#System error while enumerating the domain controllers

# Windows Domain Visibility Issues

watchfor /System error while enumerating the domain controllers/

mail myemailaddress@work.com,subject=Unable To Enumerate Domain Controllers

# NetBackup Client Service Stoppage

#watchfor /NetBackup Client Service is shutting down/

# mail myemailaddress@work.com,subject=NetBackup Client Service Stoppage

# Sun Server Kernel Panic

watchfor /reboot after panic/

mail myemailaddress@work.com,subject=Sun Server Crash - Contact Tech Svcs. On Call Immediately

# Sun Server Storage Failure

watchfor /The number of IO errors associated with a ZFS device exceeded/

mail myemailaddress@work.com,subject=Sun Server Disk Failure - Contact Tech Svcs. On Call Immediately

# Sun Server Multiple Logon Failure

watchfor /REPEATED LOGIN FAILURES ON/

mail myemailaddress@work.com,subject=Sun Server - Multiple Logon Failures

# Sun Server SCSI Bus Reset

watchfor /got external SCSI bus reset/

mail myemailaddress@work.com,subject=Sun Server - SCSI Bus Reset

# Sun Server SCSI Device Failure

watchfor /Command failed to complete...Device is gone/

mail myemailaddress@work.com,subject=Sun Server - SCSI Device Failure

****************************************************************************************

I absolutely love swatch. As you can see, I only have a few things that it is watching for, but it’s such a nice change to be alerted the second there’s an issue. Since I carry a Blackberry, I get the notifications no matter where I’m at. I suggest you read up on swatch, as there are several alert methods you can use. You can have it send emails, SMS messages to cell phones, tie it to qpage for alphanumeric pager alerts, etc. You can also have it pop messages up on the screen.

Swatch uses Perl regular expressions to parse the log entries, so it’s very easy to configure new alert watches. Imagine being alerted when specific services are restarted, servers are rebooted, users login, etc. You can report on absolutely anything that is sent to syslog. I have turned up the logging level in our domain so that I can capture all kinds of Windows logon and security events as well. You may or may not choose to do the same. Be warned that doing this nearly doubled the amount of syslog data that I capture on a daily basis.

Finally, here is the script I use to restart swatch. I had issues with the original init script getting swatch to restart cleanly, so I wrote my own.

****************************************************************************************

# !/bin/sh

# This shell script takes care of starting and stopping swatch.

# processname: swatch

RETVAL=0

test -x /usr/bin/swatch || exit 0

start(){

echo "Starting swatch"

# Spawn a new swatch program

/usr/bin/swatch -c /etc/swatchrc --tail-file=/var/log/hosts/current/* &

echo $PID

return $RETVAL

}

stop () {

# stop daemon

echo "Stopping swatch:" $PROG

# killall swatch

for i in `ps aux |grep swatch |grep -v "grep" |gawk '{print $2}'`; do kill -9 ${i}; done

return $RETVAL

}

restart () {

stop

start

RETVAL=$?

return $RETVAL

}

case "$1" in

start)

start

;;

stop)

stop

;;

restart)

restart

;;

echo "Usage: $0 {start|stop|restart}"

RETVAL=1

esac

exit $RETVAL

****************************************************************************************

SYNOPSIS

And that’s it! All in all, this is a pretty simple thing to get going in any environment. You may choose to customize it further, or scale it down from what I have provided. Whatever you decide, I hope that this tutorial was helpful to you.

2009-08-31T10:08:00.003-04:00

This fix took me forever to find, so I thought I'd just throw it on here for the masses. ...Not that many of the "masses" come to my blog.

I upgraded VMware Server on two of our Redhat 4 systems here in the last week. The first was a test run, and went flawlessly. The actual production upgrade went fine, but then I couldn't log in to the new web-based management interface for VMware.

I could open the login page by going to "https://servername:8333". Then I'd enter the username and password, and promptly be presented with the error "Web service not available." After much searching, I finally found the answer for my scenario:

Apparently, my hosts file had "localhost" pointing to the actualy IP address of the server, and not "127.0.0.1". I corrected this, and voila! I was instantly able to log in to the management interface.

That's it. Short and sweet. I hope this post helps some of you out there who are upgrading to VMware Server 2.

Disk Health Monitoring With Smartmon

2009-02-12T13:18:00.009-05:00

Today, I've decided to load the smartmontools package on one of my Solaris 10 file servers. This toolset allows an administrator to make use of the extra features in all S.M.A.R.T. capable hard drives. What I am interested in doing is configuring a storage server to run the smartd daemon and email me when a disk is throwing errors. Hopefully, this will help me to preempively replace disks before an issue arises.

Before we begin, I have to give credit to "Matty" for both of these posts: Blog O' Matty #1 and Blog O' Matty #2. Without them, I would probably still be trying to figure this out.

Here are the refined steps I used to set this up on one of my Solaris 10 storage systems. Enjoy.

Installing Smartmon on Solaris 10

Download smartmontools from here.

# wget http://downloads.sourceforge.net/smartmontools/smartmontools-5.38.tar.gz

# gunzip smartmontools-5.38.tar.gz |tar xvf -

# cd smartmontools-5.38

# ./configure --sbindir=/usr/sbin \
--sysconfdir=/etc \
--mandir=/usr/share/man \
--with-docdir=/usr/share/doc/smartmontools-5.38 \
--with-initscriptdir=/etc/init.d

# make

# su

# make install

Create three service scripts in /usr/local/bin: smartd.start, smartd.stop, and smartd.restart:

#!/bin/sh
/etc/init.d/smartd start

#!/bin/sh
/etc/init.d/smartd stop

#!/bin/sh
/etc/init.d/smartd restart

Now create a new xml file called "smartd.xml":

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle type='manifest' name='smartd'>

<service
name="application/smartd"
type="service"
version="1">

<create_default_instance enabled="true"/>

<exec_method
type='method'name='start'
exec='/usr/local/bin/smartd.start'
timeout_seconds='3'>
</exec_method>

<exec_method
type='method'
name='stop'
exec='/usr/local/bin/smartd.stop'
timeout_seconds='3'>
</exec_method>

<exec_method
type='method'
name='restart'
exec='/usr/local/bin/smartd.restart'
timeout_seconds='3'>
</exec_method>

</service>
</service_bundle>

Save the file and test it with svccfg:

# svccfg validate smartd.xml
# echo $?
0

If you get the utterly unuseful error "svccfg: couldn't parse document", use xmllint to find the offending portion:

# xmllint -valid smartd.xml

correct any errors and revalidate with svccfg.

Now import the new manifest:
# svccfg import smartd.xml

List the properties of the new service for verification:
# svccfg -s application/smartd listprop

Edit /etc/smartd.conf to your liking, so that it will run whatever tests you require for your environment. For my purposes, I simply added a line for every disk in the server:

/dev/rdsk/c1t0d0 -d scsi -o on -a
/dev/rdsk/c1t1d0 -d scsi -o on -a
/dev/rdsk/c1t2d0 -d scsi -o on -a
/dev/rdsk/c1t3d0 -d scsi -o on -a
/dev/rdsk/c3t0d0 -d scsi -S on -o on -a
/dev/rdsk/c3t1d0 -d scsi -S on -o on -a
/dev/rdsk/c3t2d0 -d scsi -S on -o on -a
/dev/rdsk/c3t3d0 -d scsi -S on -o on -a
/dev/rdsk/c3t4d0 -d scsi -S on -o on -a
/dev/rdsk/c3t5d0 -d scsi -S on -o on -a
/dev/rdsk/c3t8d0 -d scsi -S on -o on -a
/dev/rdsk/c3t9d0 -d scsi -S on -o on -a
/dev/rdsk/c3t10d0 -d scsi -S on -o on -a
/dev/rdsk/c3t11d0 -d scsi -S on -o on -a
/dev/rdsk/c3t12d0 -d scsi -S on -o on -a
/dev/rdsk/c3t13d0 -d scsi -S on -o on -a
/dev/rdsk/c3t14d0 -d scsi -S on -o on -a
/dev/rdsk/c3t15d0 -d scsi -S on -o on -a
/dev/rdsk/c5t0d0 -d scsi -S on -o on -a
/dev/rdsk/c5t1d0 -d scsi -S on -o on -a
/dev/rdsk/c5t2d0 -d scsi -S on -o on -a
/dev/rdsk/c5t3d0 -d scsi -S on -o on -a
/dev/rdsk/c5t4d0 -d scsi -S on -o on -a
/dev/rdsk/c5t5d0 -d scsi -S on -o on -a
/dev/rdsk/c5t8d0 -d scsi -S on -o on -a
/dev/rdsk/c5t9d0 -d scsi -S on -o on -a
/dev/rdsk/c5t10d0 -d scsi -S on -o on -a
/dev/rdsk/c5t11d0 -d scsi -S on -o on -a
/dev/rdsk/c5t12d0 -d scsi -S on -o on -a
/dev/rdsk/c5t13d0 -d scsi -S on -o on -a
/dev/rdsk/c5t14d0 -d scsi -S on -o on -a
/dev/rdsk/c5t15d0 -d scsi -S on -o on -a

Enable the new service and verify that it's running as expected:
# svccfg enable application/smartd
# ps -elf |grep smartd

Originally, I was going to use the "-m" function to send email alerts, but I found that smartd works quite well with syslog. Since I already have a centralized syslog server, I'll just add a swatch statement to watch for smartd entries and then send email alerts from there.

2009-01-31T09:38:00.003-05:00

Looks like Google is having issues this morning. Any search result I click on, with the exception of a Google-hosted site, is issuing a malware warning. Let's see how long it takes to resolve. The issue started for me at 9:34AM EST:

Syslog on Solaris 10 - Remote Loghosts

2009-01-28T14:15:00.002-05:00

Just a quick note here to help others that may run into the same issue. I have had a Solaris 10 FTP server running for several years. I have grossly neglected it and only recently decided to give it any attention whatsoever.

I wanted to have the server send syslog messages to a remote loghost. Normally, this is very simple. I would just add a line in /etc/hosts to define the remote loghost, and then add a line in /etc/syslog.conf to tell syslogd which events to send to the loghost.

My problem was that syslogd kept determining that the local system was "loghost", and not the server that I defined in /etc/hosts. I could verify this by killing the syslogd process and then running it manually with the "-d" parameter.

It turns out that there was another entry on the system, defining the localhost as "loghost" in the file /etc/inet/ipnodes.

I simply removed the line altogether, restarted syslogd and voila! The remote loghost started receiving messages from my estranged FTP server.

2009-01-27T09:11:00.004-05:00

Bah! Why does Sun Microsystems force the Solaris community to log into a website to download patches and patch clusters? Why can't they just have an FTP site we can get the updates from?

I constantly have problems with the sunsolve site; It's frequently slow as hell, and just throws error messages at me all the time. Today, I can't even get a patch cluster:
Meh. I know this is an unproductive post, but no one else I know will care about this problem. Maybe one of the billions out there on the Internet will feel my pain.

Update- Apparently, I'm not alone afterall:
http://ptribble.blogspot.com/2007/08/sunsolve-needs-makeover.html

Status Update

2008-12-20T10:41:00.003-05:00

It's been about a month since I implemented the new ZFS disk configuration, and it is still running quite well. There have been no new errors or issues whatsoever.

As a matter of fact, our Applications Development team apparently had to recode parts of their app to make it run slower. It seems that the jobs were finishing too quickly, and were throwing off the nightly batch process timings on the mainframe.

Oh, and as far as the early system crash is concerned, I think that it may have had something to do with Netbackup. In the dump analysis, I found that there was only one user job running, and backups were also active. I loaded the latest maintenance pack for Netbackup, and we haven't had any problems since. We've tried to recreate the issue since, but cannot- Which is great!

Kernel Panic - Oh No!

2008-11-20T10:02:00.004-05:00

Well, it appears that the newly upgraded V240 that I was so impressed with crashed last night. It came right back up and hasn't had any issues since, but the fact that it happened at all is disturbing. There was only one user connected, and one job running at the time. Backups were running too. If anyone out there is proficient with picking through dump files, here's some mdb output for you to enjoy:

# dumpadm
Dump content: kernel pages
Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/sunfire
Savecore enabled: yes
# cd /var/crash/sunfire/
# ls
bounds unix.0 vmcore.0
# mdb 0
Loading modules: [ unix genunix specfs dtrace zfs sd pcisch ip hook neti sctp arp usba fcp fctl qlc nca lofs mpt md cpc random crypto wrsmd fcip logindmux ptm ufs sppp nfs ]
> ::status
debugging crash dump vmcore.0 (64-bit) from sunfire
operating system: 5.10 Generic_137137-09 (sun4u)
panic message: BAD TRAP: type=31 rp=2a1009768e0 addr=0 mmu_fsr=0 occurred in module "unix" due to a NULL pointer dereference
dump content: kernel pages only
> ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 834712 6521 81%
Anon 97092 758 9%
Exec and libs 3492 27 0%
Page cache 3202 25 0%
Free (cachelist) 1543 12 0%
Free (freelist) 88943 694 9%

Total 1028984 8038
Physical 1025981 8015
> ::cpuinfo
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 0000183bb88 1b 1 0 105 no no t-0 2a100977ca0 sched
1 0000180c000 1d 1 0 41 yes no t-0 30014117080 sas.e9bb95
> panic_thread/J
panic_thread:
panic_thread: 2a100977ca0
> 2a100977ca0::findstack
stack pointer for thread 2a100977ca0: 2a100975d51
000002a100975e01 die+0x78()
000002a100975ee1 trap+0x9e0()
000002a100976031 ktl0+0x48()
000002a100976181 ip_wput_ioctl+0xc4()
000002a100976231 tcp_xmit_early_reset+0x6b8()
000002a100976341 tcp_xmit_listeners_reset+0x1f4()
000002a100976411 ip_tcp_input+0xaf8()
000002a1009764f1 ip_input+0xa70()
000002a100976661 putnext+0x218()
000002a100976711 ce_intr+0x764c()
000002a1009771e1 pci_intr_wrapper+0xb8()
000002a100977291 intr_thread+0x168()
> $r
%g0 = 0x0000000000000000 %l0 = 0x0000060016e5eef0
%g1 = 0x00000000000001c0 %l1 = 0x000000007be78638 ip_ire_delete
%g2 = 0x0000000000005316 %l2 = 0x000000007001ac00 ip_areq_template+0x24
%g3 = 0x000006001f298254 %l3 = 0x0000000000005000
%g4 = 0x000006001f2981f0 %l4 = 0x0000000000000006
%g5 = 0x000006001f2981f0 %l5 = 0x000000007be783a8 ip_ire_advise
%g6 = 0x0000000000000010 %l6 = 0x000000007001aca8 ip_ioctl_ftbl+0x30
%g7 = 0x000002a100977ca0 %l7 = 0x000006001f2981f0

%o0 = 0x0000000000000000 %i0 = 0x00000600421dfb00
%o1 = 0x000002a100977ca0 %i1 = 0x0000060016e01380
%o2 = 0x0000000000000001 %i2 = 0x0000060016dcd0c0
%o3 = 0x0000000000005316 %i3 = 0x0000000000000000
%o4 = 0x0000000000000000 %i4 = 0x0000060011003e48
%o5 = 0x0000000000000064 %i5 = 0x0000000000000000
%o6 = 0x000002a100976181 %i6 = 0x000002a100976231
%o7 = 0x000000007be6a120 ip_wput_ioctl+0xc4 %i7 = 0x000000007bed8b94 tcp_xmit_early_reset+0x6b8

%ccr = 0x44 xcc=nZvc icc=nZvc
%fprs = 0x00 fef=0 du=0 dl=0
%asi = 0x80
%y = 0x0000000000000000
%pc = 0x0000000001047824 mutex_enter+4
%npc = 0x0000000001047828 mutex_enter+8
%sp = 0x000002a100976181 unbiased=0x000002a100976980
%fp = 0x000002a100976231

%tick = 0x0000000000000000
%tba = 0x0000000000000000
%tt = 0x31
%tl = 0x0
%pil = 0x6
%pstate = 0x016 cle=0 tle=0 mm=TSO red=0 pef=1 am=0 priv=1 ie=1 ag=0

%cwp = 0x04 %cansave = 0x00
%canrestore = 0x00 %otherwin = 0x00
%wstate = 0x00 %cleanwin = 0x00
> 2a100977ca0::thread -p
ADDR PROC LWP CRED
000002a100977ca0 1839750 60015eee058 60011003e48
> 1839750::ptree
0000000001839750 sched
0000060013401848 fsflush
0000060013402468 pageout
0000060013403088 init
000006001bd804b8 bpbkar
00000600183879b0 bpbkar
0000030015371ab8 bpbkar
00000300228a5238 bpbkar
0000030035cae210 bpbkar
0000030035ee12a8 bpbkar
0000030034cf8668 bpbkar
000006002dfce180 bpbkar
000003001e6b4e58 bpbkar
000006001b98e4a8 java
000006001bd7e058 dtlogin
000006001b8910c0 fmd
0000060019aeec48 snmpXdmid
000006001b98d888 dmispd
00000600145bf850 vold
000006001b7d0038 snmpdx
000006001b98c048 sendmail
000006001b9f50d0 snmpd
000006001b98f0c8 sendmail
00000600147f3098 syslogd
000006001b8904a0 sshd
000006001b7d1878 automountd
000006001aa9c030 automountd
000006001993b860 smcboot
000006001993a020 smcboot
000006001aa9e490 smcboot
000006001b7d30b8 utmpd
0000060019aee028 inetd
0000030031aa5270 in.telnetd
0000030034f212b0 ksh
0000030026e2bab0 sas.e9bb95
000006001667cda8 elssrv
000006001b9f44b0 in.telnetd
0000030027c7a6c8 ksh
00000600291e0220 sas.e9bb95
000006001b5460e8 elssrv
000006001b98cc68 in.telnetd
000006001b7d2498 ksh
0000060016667990 sas.e9bb95
0000060015f00db0 elssrv
000003002b6a46a8 in.telnetd
> $c
mutex_enter+4(600421dfb00, 60016e01380, 60016dcd0c0, 0, 60011003e48, 0)
tcp_xmit_early_reset+0x6b8(7be25368, 0, 6001f2981f0, 10, 0, 0)
tcp_xmit_listeners_reset+0x1f4(6001c73da80, 14, 0, 60013130000, 60033df1d40, b88c608d)
ip_tcp_input+0xaf8(18, 60015f1ee10, 30000d98068, 60033df1d40, 0, 30000d98068)
ip_input+0xa70(60015f1ee10, 0, 0, 30000d98068, 0, 0)
putnext+0x218(600143b6ed0, 600143b6ce0, 6001c73da80, 100, 600143b6a50, 0)
ce_intr+0x764c(1069128, 0, 6001c73da80, 11999b8, 600143b6a50, 600141eb700)
pci_intr_wrapper+0xb8(60014b12420, 300000b8148, 0, 0, 60014bd9548, 0)
intr_thread+0x168(ffffffff75702bdc, ffffffff7a9263a4, 4, 0, 0, 3)

Solaris 10, ZFS, and Dell JBODs - Redux - Update

2008-11-20T08:55:00.003-05:00

As promised, here are the results of adding a second disk array to the SunFire V240. Interestingly, I didn't really see much of a performance boost by adding the second array:

The iozone charts above show the sequential read and write performance with two Dell PV220s disk arrays attached. Write performance stayed pretty much the same, and read performance only improved by about 200MB/second. Now, both of these arrays are plugged into the same SCSI controller, so either I've reached the capacity of the SCSI card or the server's PCI bus.

I think it's far more likely that the SCSI controller is simply doing all it can to keep up. I am now looking into moving the second disk array to its own SCSI controller. I expect this will yield yet another boost in performance. I'm not sure about the ramifications of trying to migrate 14 out of 28 disks in a single zpool to another controller. I'll have to research that one a bit.

New Backup System

2008-11-14T11:05:00.001-05:00

Our backup system is starting to get a little "long in the tooth", if you know what I mean. Currently we've got a single Windows 2003 server running Netbackup 5.0. It sends the incremental backups to a fibre-attached EMC CX300, and then dups them off to tape. The weekly and monthly fulls go straight to tape on an Adic Scalar 100 with 5 LTO2 drives, which is also connected via fibre.

The CX300 is no longer under warranty, and disk are going offline left and right. Dell wants $16K to renew maintenance and another few thousand to upgrade the firmware on it to the current revision. Unfortunately, with the CX300 only being used for d2d backups, it's really not worth the money to renew and update the thing.

I also hate running Netbackup on Windows. I love Netbackup, but I just feel that I could better leverage its capabilities on a Solaris system.

And finally, we spend a ton of money on offiste tape storage every year.

So to sum up, I want to:
- Get off of the CX300
- Upgrade from Netbackup 5 to Netbackup 6.5
- Use Solaris instead of Windows for the master server
- Drastically reduce offsite tape storage services

If I go with a non-sparc server, I can run Solaris on x86 and save some money. I figure something like a Dell PowerEdge 2970 or the like. Those only run about $4K.

I did some searching and found that XStore carries a 24-disk SAS/SATA JBOD chassis for about $2500. I can get some Seagate 1TB enterprise-class disks for $205 a piece. Probably less if I buy in bulk. This will set me up with a smoking fast, 24TB d2d system for under $13K. Tack on the Netbackup upgrade for another $12K and we're up to $25,000.

This takes care of The first three issues I have with the existing backup system. Now for the offsite storage. We've been with the same vendor for about four years now, and we coincidentally only have four years of backups stored there. This year will cost us a little thess than $30,000 to store our tapes offsite. For legal reasons, we need to retain seven years so we can assume that our annual spend for offsite storage will nearly double by the end of 2011. So, it's definitely worth it to look into an alternative.

To tackle this, I'm currently thinking about a software-based, block level data mirroring solution. Something like Double-Take might do the trick. We've used it in the past, but only to keep user data on two file servers synchronized over a WAN. Generic user data is a lot different from compressed d2d backup images, so I'm not sure how viable a solution like this really is.

If the synchronization does work though, then I will look at building out two of these servers with two of the SATA arrays on each one. This should provide me with the ability to store all daily incrementals as well as weekly fulls on disk. I'll only have to offsite the monthly fulls. I figure I can just send the NY tapes to VA, and the VA tapes to NY. Or I could just keep the monthlies at the existing offsite storage vendor.

Anyway, with the synchronization piece, the project cost now jumps to about $55,000. If I can manage to trade-in or sell the CX300 and reuse existing servers, then this will reduce the capital outlay even more. I would love to have an ROI of less than 12 months.

I'll post more as I refine my plan. Input is certainly welcome.

Solaris 10, ZFS, and Dell JBODs - Redux

2008-11-14T10:14:00.001-05:00

As the follow-on to my previous post, I just completed another project in which again shows that ZFS with Dell JBODs just makes sense.

Another team of SAS users has a Sunfire V240 which had about the worst disk configuration I have ever seen, running Solaris 9. The performance was absolutely awful. Here's the scenario:

- One 1GB fibre channel connection to an EMC Symmetrix 8530.
- The 8530 served up 17 concatenated disk pairs as, get this, 116 9GB disks. WTF??
- These 116 9GB logical disks were then combined into 13 RAID5 groups- Try and make sense of iostat -xn output with that crap!
- One 2GB fibre channel connection to a 4-disk (300GB, 10K) RAID5 group on an EMC CX300.

To give you an idea of the performance, here' s the iozone read/write results under that config:

350MB/sec write speed?! Blech! And the read is only a paltry 900MB/sec. With that many spindles on FC, this thing should scream. Oh, and this system is used for data mining of all things.

I was constrained by the parameters of the project as well. I couldn't buy a new server, and we were phasing out both the Symmetrix and the CX300. I needed a lot of disk and it had to perform.

I really liked the performance gain we got with the Sunfire v445 and Dell JBODs on the previously posted project and decided to go with a similar config.

So, I ran a full backup of the server and then shut everything down. Disconnected the FC cables and pulled the HBAs. Dropped in two shiny new SCSI controllers, connected the PV220s arrays and fired it up. It was only then that I realized that I had two different speed arrays. One was a U320 and one was a U160. It was do or die time, so I proceeded with just one array. If performance was poor, I could get the second array upgraded to U320 in just a few days.

I loaded Solaris 10 on the system, using the cool new ability to boot into ZFS. This time, I created a zpool with two 7-disk raidz vdevs:

zpool create sbidata raidz c3t0d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 c3t5d0 c3t8d0 raidz c3t9d0 c3t10d0 c3t11d0 c3t12d0 c3t13d0 c3t14d0 c3t15d0

I then restored the passwd, shadow & group files, the home directories, the SAS application, and the data from the old system. Then I fired up the system and ran some tests. The iozone results with just the single 14-disk JBOD were staggering:

Unbelievable. Write performance had more than tripled and read performance doubled! Keep in mind, I went from two EMC fibre channel arrays to a single, 14-disk SCSI JBOD. The previous configuration was just that bad.

Anyway, batch jobs that took 11 hours now only take 5. User driven job time has now been cut by as much as 80% in some cases and at least 66% in most. I just got the parts to upgrade the second JBOD to U320, and will make the change tomorrow morning. I will post the new iozone results when I'm done.

I can't wait to see what the performance looks like tomorrow afternoon!

Solaris 10, ZFS, and Dell JBODs

2008-11-14T09:25:00.001-05:00

For everyone else out there who is trying to do more with less, I thought I would post some of my projects that I feel really had a lot of "bang for the buck".

The first in the series is regarding Solaris 10 and my new favorite filesystem, ZFS. In Spring of '07, I was charged with migrating a team of SAS users off of a Sun V240 and onto a larger V445. Not only did they need a good amount of disk space, but performance was a critical factor. They would also likely grow about 8-12% per year.

The problem was, there just wasn't money in the budget for an expensive SAN. So, I started testing out ZFS in the lab with spare equipment and was amazed at the performance.

After enough testing, I decided to go with the Sunfire V445 and two Dell PowerVault 22os JBOD arrays. I loaded the 220s with 14 U320 146GB SCSI disks, and direct attached each array to a separate SCSI controller onthe server.

Now, at the time I was still very new to ZFS and did not choose an optimal configuration. I figured that more spindles in a raid array meant better performance, so I assigned 21 of the 28 disks to the zpool in a raidz vdev:

zpool create sbimktg raidz c2t0d0 c2t1d0 c2t2d0 c2t3d0 c2t4d0 c2t5d0 c2t8d0 c2t9d0 c2t10d0 c2t11d0 c2t12d0 c2t12d0 c2t14d0 c2t15d0 c3t0d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 c3t5d0 c3t8d0

And, voila! A shiny new ZFS raidz for the marketing folks. I then ran iozone to get an idea of performance, and things looked great:

I know, I know- With the components I used, I should be able to reconfigure and obtain much better performance than what's shown in the graphs. But, compared to what we were getting on the old server, this was a phenomenal performance boost.

I am planning on a reconfig in the near future, which will ultimately put the data on a single zpool consisting of four 7-disk raidz vdevs. This should substantially boost performance for the marketing team.

For The Love Of UNIX

More Sun Website Problems

Disk Health Monitoring With Smartmon

Syslog on Solaris 10 - Remote Loghosts

Status Update

Kernel Panic - Oh No!

Solaris 10, ZFS, and Dell JBODs - Redux - Update

New Backup System

Solaris 10, ZFS, and Dell JBODs - Redux

Solaris 10, ZFS, and Dell JBODs