NetApp CIFS and BackupExec12: What the What?

After adding our shiny, new shelf to our FAS2050A running ONTAP 7.2.5.1, I migrated all of the CIFS shares from controller A over to the new volume created on the shelf assigned to controller B. After completing migration and updating the DFS links, I let my backup administrator know to change the backups to point to controller B. However, when he tried, he kept getting credentials failed in Backup Exec, no matter what he used for credentials.

Symptoms:

Able to connect to Filer 1 via network share.
Able to connect to Filer 2 via network share.
Able to connect to Filer 1 via BackupExec (enumerated shares)
UNAble to connect to Filer 2 via BackupExec (prompted for credentials)

Solution:
Disable ndmp on filer.

If you do not have the ndmp license on Backup Exec so you are backing up the resources via Windows shares, ndmp causes Backup Exec to fail. I don't know why. It would be nice if Backup Exec would have a more informative error.

NDMP is disabled by default; however, in the process of transferring the CIFS qtrees from one controller to the other, I enabled it to get fast transfers while maintaining premissions.

Duplicate bootable BartPE USB in Ubuntu

I created this nice bootable USB using BartPE and, since then, have deleted the source off of my hard disk. Desktop Support just asked if I could copy the contents of my USB to theirs so they can enjoy all of that BartPE goodness. Trouble was, you can't copy the files and expect the USB to be bootable. To add to the fun, my USB was a 1GB stick and theirs was a 2GB.

Easiest solution: dd.

1. Plug both usb's in your computer. Allow them to mount.
2. Ensure you know which one is which. I ran df -H and could tell the 1GB (source) was /dev/sdb1 and the 2GB (destination) was/dev/sdc1
3. Dismount the disks (from command line, umount /dev/sdb1 && umount /dev/sdc1)
4. Duplicate the disks block for block: dd if=/dev/sdb of=/dev/sdc
5. **Important** After the dd command has completed, pull the usb sticks out and put the destination back in. This allows for the new fat table to be re-read.
6. Using gparted (sudo apt-get install gparted) resize the 1GB partition to the full disk size.

twitux: the goggles, they do nothing

After reloading my ubuntu (jaunty not ready, yet) I installed (via apt-get) twitux. The layout was terrible. Apparently, there was a time when graphics were not being resized before being uploaded to Amazon (from what I've read here and here). Twitux is not resizing them for us, so huge images are destroying the layout of twitux. People are asking that twitux be be modified to resize the images and give the option to not show images at all.

In further research, I came across this gem where someone had older avatars that were not being updated. The fix was to delete the avatars out of ~/.gnome2/twitux/avatars. That gave me an idea. While we wait for an update to twitux, why not overwrite the avatars with our own graphic?

Here's what I did:

1. In Gimp, create a new file, 1x1 pixels, save as ~/blank.jpg
2. Apply the script below (put it in a loop, chron, etc)
3. Close, re-open twitux

The script I use is getto simple, the graphic I created in step 1 came out to be 306 bytes, so the script overwrites anything larger than that with our blank graphic.

#!/bin/bash
for i in $( find ~/.gnome2/twitux/avatars -type f -size +307c ); do
cp ~/blank.jpg $i
done

Here is the result:

Split Tunnel VPN problems

Two high-end attorneys went out of the country to work on a case. Problem was, they were not able to use their firm laptops to VPN in. We use Cisco's IPSEC VPN on our ASA 5500's and, to allow both firm access and Internet access simultaneously, we use split tunneling. Not every router support IPSec split tunneling. The symptom is the user can connect to VPN (locked icon in taskbar) but, once connected, they can not see firm resources, nor can they see any Internet resources either. If they disconnect, Internet access is restored. We've seen this problem before primarily with AT&T's dsl routers. The problem is the cheaper router's implementation of NAT modifies the packet envelope. IPSec is very particular about packet integrity and discards any packets that appear to be tampered with. Usually, we can get the attorney to spend $50 on a D-Link wireless router and move on. The situation with the two attorneys was not going to be as easy.

I decided to utilize SSL VPN. SSL is supported by virtually every public router out there. Our Cisco ASA comes with two included SSL VPN licenses and there are two attorneys needing remote VPN access. Perfect. I've worked with SSL VPN appliances before, most do application level tunneling. To give the attorneys a similar experience (and to keep from having to determine what ports every application of ours uses) I was hoping Cisco had a full-VPN SSL solution. They did, it's called Tunnel Mode.

Using the example from Cisco's site, I was able to create a profile that auto-installed a client on the user's computer and initiated a remote connection tunnel.

One of the problems with clientless setups like WebVPN is it is easier to masquerade as a firm user. This opens up password hack attempts on the server side, and phishing attempts for the user. User's passwords, even with policies in place, are typically not that secure. If user/password were the only authentication method, it would most likely not take long for a hacker to gain access to at least one of our accounts. Also, using a man in the middle attack, a user could be entering their username and password on a page that looks like ours but is someone else's (yes, they would get a certificate error, but users don't ususally read/care about those.)

Following our "protect the user from themselves" guideline, they best solution was to use our already existing Microsoft Enterprise CA infrastructure for server and client-side authentication. This ensures the client is coming from a firm computer. The private key on the user's computer (auto-installed via GPO) is far stronger than any password a user would have and, if compromised, could be revoked almost instantly.

Now, the two attorneys need only to go to a web site in their browser and then the (ahem, clientless) client asks them to select a certificate. Once selected, the VPN "just works (TM)". Again, if you haven't done so already, I highly recommend looking into what a Certificate Authority can do for your environment. Thanks for reading.

PKI has made my life much easier

Our PKI system was originally put into place to allow for EFS on Windows XP Pro computers in a domain environment. Since then, we've utilized our PKI for 802.1x authentication (wired and wireless) on our network using EAP-TLS, email signature and encryption, HP Systems Insight Manager trusts, and other applications that have required authentication and/or client verification. Getting a PKI setup initially is very tricky, with many things that can cause the system to break, some lessons learned were:

• Set the root's CA's expiration date to something rather out there (I chose 75 years.)
• The root CA should never be on the network, not even for security updates. A VM with no network card works great for this. We only power it on once a year to publish the CRL to a floppy disk. The Sub CA stays on the network and publishes a CA (with deltas) constantly.
  
• Ensure that you have your CRL publish to multiple locations, including one off of your network completely. Our corporate web site serves this purpose.
  
• Test and document certificate revocation steps.
• Test and document certificate and document (EFS) recovery steps.
  
• To utilize Microsoft's CA capabiltites completely, your Enterprise CA needs to run on Windows 2003 Enterprise.

Again, the PKI is not easy to initially setup, but once done, adding additional things like EFS, SSL VPN, and encrypted email become simple.

Event log monitoring (how I sleep at night)

When I was first hired on as a Windows Server guy, I was initially tasked with getting Microsoft Operations Manager (MOM) working in our environment. After much work, I felt MOM 2005 had some deal-breaking issues:

1. Event log alerts were only flagged with known alerts. This does not work well when services such as Blackberry Enterprise Server (BES) have a new and undocumented alert every time they decide to stop working. At the time, a management agent was only provided by a third party and it still did not cover every critical event there was.
2. Disk space monitoring. By default, MOM expected every server to have the same free space needs (a percentage, or a hard value). Our servers had very different needs and trying to find one setting to fit all the servers simply didn't work. There was a solution written by some astute guys using vbs and custom settings per server, but it felt like a hack and the central management was no longer applicable.
3. Database management. MOM itself became a monster that required me to constantly tweak the database (due to the 1GB limitation) to keep it working. I would prefer to spend my resources on the production servers, not the watchdog.

I needed something that treated event management from the other direction: alert on all events except the ones I have predetermined to be benign. Also, the filtering needed to be unique per server. Disk space needed to have alert thresholds to also be unique per server and I needed to be able to do historical reporting on the servers so I could get an idea of when we would need to get more disk space.

The solution we found was Power Admin. It met our needs, the developers are approachable, and it seems our requests usually make it into the next version, which I find remarkable. Their pricing structure is very good, especially for SMB's who need to start small and add more servers as they go along.

I recommend their product highly. If you find that it meets your needs as well, drop me a comment and let me know. Thanks for reading.

Windows GPO and trusted Add-Ons

Starting with WinXP SP2, Microsoft now has a feature to only allow explicitly declared Add-Ons in Internet Explorer. For us, this is a great feature. We already have an application whitelist program (Websense Client Policy Manager) that blocks unknown executables, but add-ons such as IE toolbars can still get through. Having control over the IE add-ons completes our goal to "protect the users from themselves."

The steps for implementing Add-on blocking can be found via Google search, so I will not go into them here. For information on initial implementation can be found at How to manage Internet Explorer add-ons in Windows XP Service Pack 2.

At first, it seems deceptively simple. Find a computer with only the add-ons you want (Java, Flash, etc.) and get the CLSID's off of it by clicking manage add-ons in IE, right clicking the column select ClassID so you can see the CLSID, and add those to your GPO whitelist of allowed add-ons, call that project done. Turns out, there are many more add-ons you can not see in add-on management that will not run once you enable the option to only allow whitelisted Add-Ons.

So, here's the symptom: After whitelisting every add-on you can see in the add-on manager, when you attempt to go to certain websites (OWA, Sharepoint, Carpe Diem Web, Java VM test, etc) you will see the blocked add-on icon . You double click the icon to open the add-on manager, yet every add-on still shows as enabled. This affects IE6, IE7, and IE8. IE8's new add-on manager looks nicer, but for what we are trying to do, it offers no more functionality than the others.

MS KB article 555235 is a great place to start to add some of those hidden Add-Ons, most notably, javascript and vbscript. The XML Dom and XML Http add-ons are also important in OWA, else you will have a blank pane where all of your inbox items should be.

Unfortunately, while that should get you a good head start, MS does not list all of the hidden add-ons, and they do not give us an easy method for determining the CLSID for ourselves. MS says you can find the CLSID by looking at the object tag in the html source code. However, on all of the web sites I was testing, not one had an object tag or a CLSID I could get.

To determine what the clsid of the blocked add-on is, you will need a clean (free of malware) pc and a windows account that is part of the add-on blocking GPO.

1. Log onto the clean pc using the windows account
2. Run regedit, browse to HKCU\Windows\CurrentVersion\Ext.
3. Ensuring all IE windows are closed, delete the Settings and Stats keys (they will recreate)
4. Log off / Log back on
5. Open IE and browse to the site that causes the add-on block icon to appear
6. Run regedit, browse to HKCU\Windows\CurrentVersion\Ext\Stats
7. For each CLSID, open the iexplore key, if there is a dword value of "Blocked", then that is one of the CLSID's you need to add to your GPO's whitelist.
   
8. Repeat step 7 until you have found all blocked add-ons.

Once added to the gpo, refresh the user's gpo via gpupdate /force, or just log off and log back on. Attempting to visit that site again should have no problems.

If this has helped you, please add a comment saying so. Thanks for reading.

Hello World.

First post, so I will try not to waste your time. The purpose of this blog is to write about problems that I was not able to find the solution to on Google. I am a Systems Administrator of a law firm in Dallas, TX. We run a fairly typical setup (VMWare, Microsoft, Dell, HP, NetApp, Cisco, etc) so I suspect others may benefit from this as well. Hopefully, Google will pick this up.