Single Outlook client can't connect; Exchange errors!

A few days ago I encountered a strange Exchange / Outlook problem.  One of our Outlook 2003 installations couldn't connect to our Exchange 2007 server, no matter what we tried.  The user's PC was able to ping the server, and even connect via web-mail, but no Outlook!  The user was getting all sorts of connectivity errors in the Outlook client itself, primarily:

Unable to open your default e-mail folders. The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server is down for maintenance.

After hours of troubleshooting, I noticed that a specific event log error kept popping up on the Exchange server when we would try to connect:

ComputerName= Logfile=Application Type=Error SourceName=MSExchangeIS Category=6 CategoryString=General EventCode=9646 EventID=-1073338962 TimeGenerated=20100426134708.000000-420 TimeWritten=20100426134708.000000-420 Message=Mapi session "e49d45e3-baf2-48f6-ba0c-87b944a75f88" exceeded the maximum of 32 objects of type "session".

It turns out that the Outlook 2003 client previously opened a multiple sessions with the server, hit the 32 session limit, then crashed or otherwise left the connections hanging open, and this was preventing all new connection attempts from succeeding.

Restarting the Exchange services would have fixed this problem, but since this would have disconnected hundreds of clients (this was during the day), restarting services wasn't an option.

It turns out that the hung connections can be closed by the Windows Sysinternals tool called TCPView (LINK).  Here is how to close the connections:

1. Open the Exchange Management Shell (I'm assuming you're running Exchange 2007), and run this command (USERNAME should be the user attempting to connect via Outlook):
• Get-LogonStatistics USERNAME | Format-Table username,logontime,clientipaddress
2. This will output all 32 connections.  Make a note of the IP address(es) that the hung connections were initiated from.
3. Download TCPView (link above) and run it on the Exchange server to which the Outlook client is failing to connect.  If you have a busy server it may take a few minutes for TCPView to gather all the info.  You can turn off the refresh by selecting View --> Update Speed --> Paused.
4. Click Options --> uncheck Resolve Addresses.
5. Sort first by Remote Address, then by Process, and type "ST" in the window to scroll down to the "store.exe" process connections.
6. Find the connections to the store.exe process from the IP address(es) you noted in step 2 above.  Select all of those connections, right click, and click Close Connection.
7. Run the command in step 1 above again in the Exchange Management Shell to verify that the user has no more open connections.
8. The Outlook client should now be able to re-connect to the Exchange server successfully.  You may run into this issue again if you do not resolve whatever was causing the excess connections to begin with.

Integrating VMware VDR with Backup Exec

VMware VDR is working properly for our on-site backups and data recovery (others have had a lot of problems with it, but it works for us), and I needed to figure out a simple way to get our backups off disks and onto tape for off-site storage.  Although is possible to replicate the VDR backups for off-site storage if you are using a SAN with replication technology, I can’t allocate enough space on our SAN right now to store/replicate the backups, so that is not an option.

Our VDR appliance is sending backups to a simple NTFS share on our backup server.  The backup server has Symantec Backup Exec (2010) and an attached LTO4 tape library that I will use copy the VDR backups to tape.  The task seems pretty straightforward, except for the fact that the VDR appliance needs to be powered off in order to ensure valid backups.  You don't want VDR to start an integrity check or backup job in the middle of your copy to tape.

The first thing I thought of was to configure a scheduled task in vCenter to shut down the VDR appliance in the morning and turn it back on at night (presumably after the tape backup has completed).  This would technically work, but I don't like having VDR powered off all day.  It's a pain to power it on each time you want to check settings or to let the integrity check run. 

The solution I came up with was to add a pre-command and a post-command to the Backup Exec job to power off and power on (respectively) the VDR appliance.  The first thing I had to do was install the vSphere CLI on the backup server (I installed it to C:\vCLI, for ease of access).  Once this is installed, you'll need to create batch files for the pre/post commands (I'm not sure why, but it doesn't work if you put the commands directly into Backup Exec), then add the proper pre/post commands into Backup Exec after verifying that they do what you want (shut down and restart the VDR appliance):

1. Install the vSphere CLI on the server that runs Backup Exec.
2. Run the following command from the vSphere CLI:
• C:\vCLI\bin>vmware-cmd.pl -H ESXHOST -U USERNAME -l
3. Where ESXHOST is the ESX/ESXi host that the VDR appliance is running on and USERNAME is the account that has access to this ESX server (usually "root").
4. The previous command will give you a list of vmx files.  Copy the line that lists the VDR appliance's vmx file.  It will probably look something like this:
• /vmfs/volumes/4bbb70ad-10e222d0-4d79-001018645db2/VDR/VDR.vmx
5. This gives you all the info you need to create your pre/post batch files.  They will be (starting with the pre-command batch file):
• C:\vCLI\bin\vmware-cmd.pl -H ESXHOST -U USERNAME -P PASSWORD "VMX-FILE-PATH" stop soft
• C:\vCLI\bin\vmware-cmd.pl -H ESXHOST -U USERNAME -P PASSWORD "VMX-FILE-PATH" start
6. Test these batch files before starting your backup job, and make sure that they work properly!  The first one should gracefully shut down your VDR before the backup job begins, and the second one will power it back on when the backup completes.
7. Add the batch files you created to the pre/post command fields inside your Backup Exec job.
8. Now theVDR appliance will only be offline during the actual backup process!