totally cheating here

Google search will certainly reveal this one, but if you are looking into upgrading (or building) your server infrastructure and heard about the benefits of blades instead of standalone rack servers, then Blades Made Simple (TM) is one of the sites that will take a lot of the guesswork out of why choose blades (and WHICH blades to choose).

This week, they are celebrating their one year anniversary with a free giveway. What they are giving away is anyone's guess (I vote for a Cisco UCS packed with B230's maxed to the gills in RAM). At a minimum, those lucky few selected can expect a bag of m&m's to be sent to them gratis.

If I win the m&m's, I will give it to a worthy cause. If I win a UCS, I am keeping it to myself. Sorry, my charitable nature only takes me so far.

Oh, and technically, they are calling it a "Bloggiversary", but I can't seem to call it that. It's like when the smurfs would take arbitrary words and replace it with smurf to give it more smurfiness.

Congrats, Blades Made Simple (TM) on the one year, keep up the informative posts.

NetApp, updates, and NDMP: what is this I don't even

So after upgrading from ONTAP 7.3.1 to 7.3.2P3, I get an alert from our NOC saying NDMP backups are failing. Passwords had not been changed on any accounts since the last good backup. I checked the changelogs between 7.3.1 and 7.3.2P3 and did not find anything related to NDMP.


Symptom:
NDMP backups fail, citing authentication errors.

An attempt to re-generate an NDMP password yields this error:


filer01> ndmpd password ndmpfiler01
Cannot generate NDMP password.

filer01> version
NetApp Release 7.3.2P3: Fri Dec 11 17:58:49 PST 2009


Cause:
Sometime between then and now, a new capability was added: login-ndmp. If an account does not have that capability, ndmp logins and password generations will fail.


Notes:
We have a least-privilege account used for backing up the filers. A role was created called ndmp_role with the following capabilities: cli-ndmpcopy*, cli-ndmpd* which worked in 7.3.1. In 7.3.2P3 (don't know which actual version introduced this), the capability was added. A role called backup was created with the login-ndmp capability. A group called "Backup Operators" was added that includes that role.



Solution:
Add your least-privilege ndmp account to the "Backup Operators" group. Once added, you can again perform an ndmp password command against that account.

NetApp CIFS and BackupExec12: What the What?

After adding our shiny, new shelf to our FAS2050A running ONTAP 7.2.5.1, I migrated all of the CIFS shares from controller A over to the new volume created on the shelf assigned to controller B. After completing migration and updating the DFS links, I let my backup administrator know to change the backups to point to controller B. However, when he tried, he kept getting credentials failed in Backup Exec, no matter what he used for credentials.

Symptoms:

Able to connect to Filer 1 via network share.
Able to connect to Filer 2 via network share.
Able to connect to Filer 1 via BackupExec (enumerated shares)
UNAble to connect to Filer 2 via BackupExec (prompted for credentials)

Solution:
Disable ndmp on filer.

If you do not have the ndmp license on Backup Exec so you are backing up the resources via Windows shares, ndmp causes Backup Exec to fail. I don't know why. It would be nice if Backup Exec would have a more informative error.

NDMP is disabled by default; however, in the process of transferring the CIFS qtrees from one controller to the other, I enabled it to get fast transfers while maintaining premissions.

Duplicate bootable BartPE USB in Ubuntu

I created this nice bootable USB using BartPE and, since then, have deleted the source off of my hard disk. Desktop Support just asked if I could copy the contents of my USB to theirs so they can enjoy all of that BartPE goodness. Trouble was, you can't copy the files and expect the USB to be bootable. To add to the fun, my USB was a 1GB stick and theirs was a 2GB.

Easiest solution: dd.

1. Plug both usb's in your computer. Allow them to mount.
2. Ensure you know which one is which. I ran df -H and could tell the 1GB (source) was /dev/sdb1 and the 2GB (destination) was/dev/sdc1
3. Dismount the disks (from command line, umount /dev/sdb1 && umount /dev/sdc1)
4. Duplicate the disks block for block: dd if=/dev/sdb of=/dev/sdc
5. **Important** After the dd command has completed, pull the usb sticks out and put the destination back in. This allows for the new fat table to be re-read.
6. Using gparted (sudo apt-get install gparted) resize the 1GB partition to the full disk size.

twitux: the goggles, they do nothing

After reloading my ubuntu (jaunty not ready, yet) I installed (via apt-get) twitux. The layout was terrible. Apparently, there was a time when graphics were not being resized before being uploaded to Amazon (from what I've read here and here). Twitux is not resizing them for us, so huge images are destroying the layout of twitux. People are asking that twitux be be modified to resize the images and give the option to not show images at all.

In further research, I came across this gem where someone had older avatars that were not being updated. The fix was to delete the avatars out of ~/.gnome2/twitux/avatars. That gave me an idea. While we wait for an update to twitux, why not overwrite the avatars with our own graphic?

Here's what I did:

1. In Gimp, create a new file, 1x1 pixels, save as ~/blank.jpg
2. Apply the script below (put it in a loop, chron, etc)
3. Close, re-open twitux

The script I use is getto simple, the graphic I created in step 1 came out to be 306 bytes, so the script overwrites anything larger than that with our blank graphic.

#!/bin/bash
for i in $( find ~/.gnome2/twitux/avatars -type f -size +307c ); do
cp ~/blank.jpg $i
done

Here is the result:

Split Tunnel VPN problems

Two high-end attorneys went out of the country to work on a case. Problem was, they were not able to use their firm laptops to VPN in. We use Cisco's IPSEC VPN on our ASA 5500's and, to allow both firm access and Internet access simultaneously, we use split tunneling. Not every router support IPSec split tunneling. The symptom is the user can connect to VPN (locked icon in taskbar) but, once connected, they can not see firm resources, nor can they see any Internet resources either. If they disconnect, Internet access is restored. We've seen this problem before primarily with AT&T's dsl routers. The problem is the cheaper router's implementation of NAT modifies the packet envelope. IPSec is very particular about packet integrity and discards any packets that appear to be tampered with. Usually, we can get the attorney to spend $50 on a D-Link wireless router and move on. The situation with the two attorneys was not going to be as easy.

I decided to utilize SSL VPN. SSL is supported by virtually every public router out there. Our Cisco ASA comes with two included SSL VPN licenses and there are two attorneys needing remote VPN access. Perfect. I've worked with SSL VPN appliances before, most do application level tunneling. To give the attorneys a similar experience (and to keep from having to determine what ports every application of ours uses) I was hoping Cisco had a full-VPN SSL solution. They did, it's called Tunnel Mode.

Using the example from Cisco's site, I was able to create a profile that auto-installed a client on the user's computer and initiated a remote connection tunnel.

One of the problems with clientless setups like WebVPN is it is easier to masquerade as a firm user. This opens up password hack attempts on the server side, and phishing attempts for the user. User's passwords, even with policies in place, are typically not that secure. If user/password were the only authentication method, it would most likely not take long for a hacker to gain access to at least one of our accounts. Also, using a man in the middle attack, a user could be entering their username and password on a page that looks like ours but is someone else's (yes, they would get a certificate error, but users don't ususally read/care about those.)

Following our "protect the user from themselves" guideline, they best solution was to use our already existing Microsoft Enterprise CA infrastructure for server and client-side authentication. This ensures the client is coming from a firm computer. The private key on the user's computer (auto-installed via GPO) is far stronger than any password a user would have and, if compromised, could be revoked almost instantly.

Now, the two attorneys need only to go to a web site in their browser and then the (ahem, clientless) client asks them to select a certificate. Once selected, the VPN "just works (TM)". Again, if you haven't done so already, I highly recommend looking into what a Certificate Authority can do for your environment. Thanks for reading.

PKI has made my life much easier

Our PKI system was originally put into place to allow for EFS on Windows XP Pro computers in a domain environment. Since then, we've utilized our PKI for 802.1x authentication (wired and wireless) on our network using EAP-TLS, email signature and encryption, HP Systems Insight Manager trusts, and other applications that have required authentication and/or client verification. Getting a PKI setup initially is very tricky, with many things that can cause the system to break, some lessons learned were:

• Set the root's CA's expiration date to something rather out there (I chose 75 years.)
• The root CA should never be on the network, not even for security updates. A VM with no network card works great for this. We only power it on once a year to publish the CRL to a floppy disk. The Sub CA stays on the network and publishes a CA (with deltas) constantly.
  
• Ensure that you have your CRL publish to multiple locations, including one off of your network completely. Our corporate web site serves this purpose.
  
• Test and document certificate revocation steps.
• Test and document certificate and document (EFS) recovery steps.
  
• To utilize Microsoft's CA capabiltites completely, your Enterprise CA needs to run on Windows 2003 Enterprise.

Again, the PKI is not easy to initially setup, but once done, adding additional things like EFS, SSL VPN, and encrypted email become simple.