Exchange 2003 Version Store Crash!

As I walked in the door this morning, our Exchange 2003 mail server greeted me with a massive failure.  The server's informaiton store service was frozen, and users were getting all sorts of bizarre errors when launching Outlook or trying to connect via webmail.

On the mail server, we noticed the following event log entries shortly before the store crashed:


Event ID 218 (Symantec Mail Security for Exchange):

The attachment "--FILENAME--" located in message with subject "--REMOVED--", located in --USERNAME--/Inbox has violated the following policy settings:

Scan: Auto-Protect
Rule: Unscannable File Rule
The following actions were taken on it:
The attachment "--FILENAME--" was Quarantined for the following reason(s):
Scan Engine Error. The maximum cumulative file size within a compressed file has been exceeded

Event ID 218 (Symantec Mail Security for Exchange):

The attachment "--FILENAME--" located in message with subject "--SUBJECT--", located in --USERNAME--/Inbox has violated the following policy settings:

Scan: Auto-Protect
Rule: Unscannable File Rule
The following actions were taken on it:
The attachment "--FILENAME--" was Quarantined for the following reason(s):
Timeout occurred while scanning this item

Event ID 9791 (MSExchangeIS Mailbox Store / Background Cleanup):

Cleanup of the DeliveredTo table for database 'First Storage Group\Mailbox Store (--SERVERNAME--)' was pre-empted because the database engine's version store was growing too large. 0 entries were purged.
 
The store finally crashed when this event log error was recorded:
 
Event ID 623 (ESE / Transaction Manager):
Information Store (10180) First Storage Group: The version store for this instance (0) has reached its maximum size of 155Mb. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.

Possible long-running transaction:
SessionId: 0x7FD31E20
Session-context: 0x00000000
Session-context ThreadId: 0x00000EB0
Cleanup: 1

Here is what happened:

One of our users was sent a few 200MB-ish

Rebooting the server fixed the problem temporarily, but within a few minutes we began receiving the same event log errors (minus the ESE error) again, leading me to believe that the store was on its way to another spectacular crash.

In order to remove these messages that were causing us so much grief, I used a utility called MFCMapi.exe.  It is extremely useful for multiple back-end Exchange tasks, and can be downloaded here.  You have to set up an Outlook profile for an Exchange Administrator on your PC, then launch MFCMapi.exe:

• Click Session --> Logon only (Does not display stores).
• Select the profile of the Exchange Administrator and click OK.  Nothing will change on the screen; that's okay.
• Click MDB --> Get Mailbox Table...
• Type in the username/password of the Exchange Administrator you created the outlook profile for, if required.
• Leave the defaults in the next window and click OK.
• This will pull up a list of all the mailboxes on the server you set up the Outlook profile to connect to.
• Find and double-click the problem user (identified in my case by the Symantec Mail Security for Exchange event log errors).
• Expand Root Container --> Top of Information Store --> Double-click the folder that contains the problem email.
• You can sort the contents in the window that appears by size.  This is the easiest way to find large emails.  Once I found the 200Mb email that was causing the problem I right clicked it and selected Delete Attachments...
• Leave the pop-up blank to delete all attachments in the selected email.

After taking the steps listed above I didn't have any further issues.  I am still not sure how a 200Mb email ended up in our mail store, however...  Let me know if you have any ideas!