Complete Frame Relay Lab Setup with T1 WICs

I set up a very simple frame-relay Cisco lab today using the following equipment:

Cisco 1841 router with a WIC 1DSU-T1 V2 card (acting as DTE #1 - TOP)
Cisco 1841 router with a VWIC2-2MFT-T1/E1 card (acting as DTE #2 - SMALL)
Cisco 3825 router with a VWIC2-2MFT-T1/E1 card (acting as DTE #3 - BOTTOM)
Cisco 3640 router with a few WIC 1DSU T1 card (acting as DCE - frame relay switch - FRSWITCH)

NOTE!  There are no serial cables involved here.  These are all T1 WICs with RJ45 connections.  See this post if you need to create T1 crossover cables for this setup.  I had a lot of trouble setting this up properly due to lack of coherent information online regarding the T1 card configuration on the router acting as a frame relay switch (not to mention it was my first attempt at setting up a frame relay network).

DOUBLE NOTE!!  These exact instructions won't work using serial cables!  If you are using serial cables, there are plenty of other instructions out there to follow.

First of all, here is the diagram of what I ended up creating:

The router named FRSWITCH acts as a frame relay switch (the telco's network), and the other two routers act as two endpoints we need to connect via frame relay!  I left the switch connected to BOTTOM from the last lab I created; it doesn't affect this configuration at all.

I'm going to assume all the cables are made, tested, and connected properly as shown in the diagram above.

Let's start by programming the FRSWITCH router:

FRSWITCH>enable
FRSWITCH#configure terminal
FRSWITCH(config)#frame-relay switching
FRSWITCH(config)#interface s0/0
FRSWITCH(config-if)#encapsulation frame-relay
FRSWITCH(config-if)#no ip mroute-cache
FRSWITCH(config-if)#frame-relay intf-type dce
FRSWITCH(config-if)#frame-relay route 100 interface s0/1 101
FRSWITCH(config-if)#frame-relay route 102 interface s3/0 101
----- This last command tells the router to take frames with DLCI 102 and send them out s3/0 re-tagged with DLCI 101 (so that the destination knows where they came from).  These two commands, along with their counterparts a few lines down, actually create the PVCs.


FRSWITCH(config-if)#no shutdown
FRSWITCH(config-if)#interface s0/1
FRSWITCH(config-if)#encapsulation frame-relay
FRSWITCH(config-if)#frame-relay intf-type dce
----- This command doesn't have anything to do with clocking or serial cables - it just affects LMI functions.

FRSWITCH(config-if)#frame-relay route 101 interface s0/0 100
FRSWITCH(config-if)#frame-relay route 102 interface s3/0 100
FRSWITCH(config-if)#no shutdown

NOTE!  At no point in the FRSWITCH configuration do you have to specify service-module T1 settings.

Next, I set up BOTTOM as a DTE:

BOTTOM>enable
BOTTOM#configure terminal
BOTTOM(config)#card type t1 0 0
BOTTOM(config)#controller t1 0/0/0
BOTTOM(config-controller)#channel-group 0 timeslots 1-24 speed 64
BOTTOM(config-controller)#interface s0/0/0:0
BOTTOM(config-if)#encapsulation frame-relay
BOTTOM(config-if)#ip 192.168.0.1 255.255.255.0
BOTTOM(config-if)#no shutdown


That finishes up the basic frame-relay setup on BOTTOM.  Next I did the same thing to SMALL, just changing the IP address:


SMALL>enable

SMALL#configure terminal

SMALL(config)#card type t1 0 0

SMALL(config)#controller t1 0/0/0

SMALL(config-controller)#channel-group 0 timeslots 1-24 speed 64

SMALL(config-controller)#interface s0/0/0:0

SMALL(config-if)#encapsulation frame-relay

SMALL(config-if)#ip address 192.168.0.2 255.255.255.0

SMALL(config-if)#no shutdown

Follow the steps above again on TOP, and all 3 routers should have no trouble pinging across the 192.168.0.0/24 network!  I also configured the Ethernet interfaces shown on the routers, then configured EIGRP, allowing communication between all subnets on the diagram, but I won't cram that into this post.

NOTE!  You can see details on how Frame Relay is configured by running "show frame-relay pvc" or "show frame-relay map" on any of the DTE routers.

Let me know if you run into any problems or have any questions - hopefully this helps document the process of using the T1 WICs for frame relay lab connections.

OSPF on 3 routers

The next step for my 3 router (and 1 switch) Cisco lab is to set up a routing protocol so that all of the subnets on the network can communicate.  Here is the current setup (see my last post for setup details):

Right now, if I try to ping TOP's E0/0 interface (172.16.10.1) from SWITCH, the ping will fail, even after setting the IP DEFAULT GATEWAY 172.16.30.1 global configuration command on SWITCH in order to set BOTTOM as it's default gateway.  The ping fails when the packet reaches BOTTOM and BOTTOM and none of BOTTOM's routes include 172.16.10.1.

Let's look at BOTTOM's routes as they stand:

BOTTOM>SHOW IP ROUTE


Gateway of last resort is not set


   172.16.0.0/24 is subnetted, 1 subnets
C     172.16.30.0 is directly connected, G0/0
   192.168.1.0/24 is variably subnetted, 4 subnets, 2 masks
C     192.168.1.9/32 is directly connected, S0/0/1:0
C     192.168.1.8/30 is directly connected, S0/0/1:0
C     192.168.1.0/30 is directly connected, S0/0/0:0
C     192.168.1.2/32 is directly connected, S0/0/0:0

So, you can see from the routing table that there is no way a ping going through BOTTOM is going to reach the 172.16.10.1, because BOTTOM doesn't know where to send it!

Now, let's enable the a very basic instance of the OSPF routing protocol on BOTTOM and TOP so they will know about each other's connected subnets.

BOTTOM>enable
BOTTOM#configure terminal
BOTTOM(config)#router ospf 1
----- This command tells the router that you want to configure instance #1 of OSPF

BOTTOM(config-router)#network 0.0.0.0 255.255.255.255 area 0
----- This tells the router to include ALL attached interfaces in the OSPF topology.  I'm not going to go into the details here, but you can use the network command to specify only interfaces with specific subnets attached to be included in the OSPF topology.


BOTTOM(config-router)#area 0 authentication message-digest
----- This sets all interfaces encompassed by the previous command to use MD5 authentication for OSPF communication.


BOTTOM(config-router)#do show ip ospf interface
----- This will list all interfaces encompassed by the OSPF network command.  We will need to set the MD5 password on each of these interfaces:


BOTTOM(config-router)#interface s0/0/1:0
BOTTOM(config-if)#ip ospf message-digest-key 1 md5 c3s76k
----- This bizarre command sets the first OSPF MD5 key to "c3s76k".  You'll need to run the same command on any interface that connects to another router running OSPF in this environment:

BOTTOM(config-if)#interface s0/0/0:0

BOTTOM(config-if)#ip ospf message-digest-key 1 md5 c3s76k

That completes the OSPF configuration for BOTTOM.  Notice, nothing has changed in the routing table.  You'll have to perform the same steps on TOP and SMALL before your OSPF environment will be properly configured!  Make sure to use the exact same area number (0) and MD5 key, or your router's won't communicate OSPF information.

After running these commands on TOP and SMALL, here is BOTTOM's new routing table:

BOTTOM>SHOW IP ROUTE

Gateway of last resort is not set

   172.16.0.0/24 is subnetted, 3 subnets

C     172.16.30.0 is directly connected, G0/0

O     172.16.20.0 [110/66] via 192.168.1.9 ... S0/0/1:0

O     172.16.10.0 [110/75] via 192.168.1.2 ... S0/0/0:0

   192.168.1.0/24 is variably subnetted, 4 subnets, 2 masks

C     192.168.1.9/32 is directly connected, S0/0/1:0

C     192.168.1.8/30 is directly connected, S0/0/1:0

C     192.168.1.0/30 is directly connected, S0/0/0:0

C     192.168.1.2/32 is directly connected, S0/0/0:0

O     192.168.1.4/30 [110/130] via 192.168.1.9 ... S0/0/1:0

                     [110/130] via 192.168.1.2 ... S0/0/0:0

I have bolded the routes added by OSPF.  You'll notice that there are two equal-cost routes to 192.168.1.4/30, because the calculated distance to the serial link between TOP and SMALL is exactly the same regardless of whether you go through TOP or SMALL to get there.

You should now be able to successfully ping 172.16.10.1 from SWITCH, because your routers have routes there and back!

Best of all, even if you lose one of the serial links, all of your subnets will be able to talk to each other once OSPF converges (convergence takes less than 5 seconds in this network).

Up next, EIGRP!

Cisco Training Lab Network

I set up a Cisco equipment training lab for myself today using some spare Cisco equipment.  The diagram below outlines the basics of the network that I set up.  You can use this information as a guide for how to set up a basic Cisco training lab!  I'll be doing some more interesting stuff with this setup as time goes on.

TOP is a Cisco 3640 with a few WIC 1DSU T1 controller cards installed.

BOTTOM is a Cisco 3825 router with a few VWIC2-2MFT-T1/E1 controller cards installed.

SMALL is a Cisco 1841 router with one VWIC2-2MFT-T1/E1 controller card installed.

SWITCH is a Cisco Catalyst 2950; a 24 port fast Ethernet switch.

In order to connect the routers back-to-back without an ISP between them, just create some cables that switch wires 1 & 2 with 4 & 5 on one end (normal straight cable on the other end).  If you're comfortable creating straight-through cat5/cat6 cables, just create a cable with OW-O-GW-Bl-BlW-G-BrW-Br on one end and Bl-BlW-GW-OW-O-G-BrW-Br on the other end.  The reason behind this cable wiring is that both routers will be sending on wires 4 & 5 and listening on wires 1 & 2.  Therefore, you must swap these two pairs so that when router 1 sends on 4 & 5, the signal will end up on router 2's listening wires, 1 & 2.

Depending on the router, you may have to set up the controller module before a serial interface even appears:

BOTTOM>enable

BOTTOM#configure terminal

BOTTOM(config)#card type t1 0 0

----- This tells the router that you have a T1 card in slot 0/0. The card's amber light(s) will probably come on now.

BOTTOM(config)#controller t1 0/0/0

BOTTOM(config-controller)#framing esf

BOTTOM(config-controller)#linecode b8zs

BOTTOM(config-controller)#clock source internal

BOTTOM(config-controller)#channel-group 0 timeslots 1-24 speed 64

----- The command above will cause the router to create a serial interface.

BOTTOM(config-controller)#interface s0/0/0:0

BOTTOM(config-if)#encapsulation ppp

BOTTOM(config-if)#ip address 192.168.1.1 255.255.255.252

BOTTOM(config-if)#no shutdown

----- The mask of 255.255.255.252 (192.168.1.1/30) breaks the 192.168.1.0/24 (255.255.255.0) subnet into many smaller subnets with 2 usable IPs each - perfect for point-to-point links!  The 192.168.1.0/30 subnet consists of the following IPs:

-- 192.168.1.0 (network IP)

-- 192.168.1.1 (usable IP - BOTTOM router)

-- 192.168.1.2 (usable IP - TOP router)

-- 192.168.1.3 (broadcast IP)

If you're not comfortable with subnetting information like this, read up on it!  It's very important knowledge for anyone working on routers and switches.

The commands listed above are all that are required (and some are optional) on this router to set up a basic point-to-point interface.  Now, we need to set up the interface on one of our other routers, TOP, which requires a slightly different method because of the different interface card used:

TOP>enable

TOP#configure terminal

TOP(config)#interface s0/0

TOP(config-if)#service-module t1 clock source internal

TOP(config-if)#service-module t1 framing esf

TOP(config-if)#service-module t1 linecode b8zs

TOP(config-if)#service-module t1 timeslots 1-24 speed 64

----- The last 4 commands replace the card and controller configuration commands used on BOTTOM, the last router we configured.

TOP(config-if)#encapsulation ppp

TOP(config-if)#ip address 192.168.1.2 255.255.255.252

TOP(config-if)#no shutdown

We should now have a working, usable point-to-point link between BOTTOM and TOP, as long as you created the cable properly and plugged it into the correct interface on each router.  Try pinging between the routers:

TOP#ping 192.168.1.1

Sending 5, 100-byte ICMP Echos to 192.168.1.1, Timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms

Now that you have this link configured, it should be a piece of cake for you to configure the remaining serial links and Ethernet interfaces to complete the network in the picture at the top of this post.

Good luck!  Let me know if you run into any problems!

Windows 7 64-bit: Full-screen 32-bit DOS program

One of the old MS-DOS programs that I need to use won't run on a 64-bit operating system, and I run Windows 7 64-bit as my desktop.  I tried changing all the compatibility settings, but nothing would work.  The next logical step (in my mind) was to run it in one of our Windows XP virtual machines.  Unfortunately, remote desktop (aka terminal services) doesn't have the capability to load full-screen MS-DOS fonts due to performance issues, and this causes the following error message:

The video device failed to initialize for fullscreen mode. Choose 'Close' to terminate the application.

So using remote desktop to access a 32-bit virtual machine was out of the question.  The next idea that came to mind was using a feature of Windows 7 called Windows XP Mode.  This simply creates a virtual machine on your Windows 7 desktop that you can use to run legacy applications.  It can be downloaded here.  Make sure you remember the password you create, because even if you check the box for automatic log in, you'll need it in a few steps.

Once I got this up and running (it's pretty buggy), I copied my MS-DOS program onto the virtual XP machine and tried to run it.  Guess what? Same error!

The video device failed to initialize for fullscreen mode. Choose 'Close' to terminate the application.

This seemed bizarre to me, so I changed the only readily-available setting for the VM:  I clicked on Tools --> Disable Integration Features.  After doing this, the program ran perfectly!

What a pain, just for some old DOS program, huh?

32/64-bit NTPRINT.INF issues

When I tried to add 32-bit drivers to our 64-bit Windows Server 2008 print server (for sharing purposes), the driver installation process kept asking for an x86 (32-bit) NTPRINT.INF file.  I'm guessing it doesn't have this because it is a 64-bit server, but when I try to point it to the NTPRINT.INF file on a 32-bit server it still doesn't like it.

The fastest solution is as follows:

• Log in as a print server Administrator on a 32-bit server (a 32-bit client machine might work also). 
• Browse to your print server's UNC path (\\printserver).
• Open the Printers and Faxes share.
• Right click on the printer you wish to add 32-bit drivers to and click Properties.
• From this point, add the drivers just as you normally would.  It shouldn't give you any NTPRINT.INF issues!

Exchange 2003 Version Store Crash!

As I walked in the door this morning, our Exchange 2003 mail server greeted me with a massive failure.  The server's informaiton store service was frozen, and users were getting all sorts of bizarre errors when launching Outlook or trying to connect via webmail.

On the mail server, we noticed the following event log entries shortly before the store crashed:


Event ID 218 (Symantec Mail Security for Exchange):

The attachment "--FILENAME--" located in message with subject "--REMOVED--", located in --USERNAME--/Inbox has violated the following policy settings:

Scan: Auto-Protect
Rule: Unscannable File Rule
The following actions were taken on it:
The attachment "--FILENAME--" was Quarantined for the following reason(s):
Scan Engine Error. The maximum cumulative file size within a compressed file has been exceeded

Event ID 218 (Symantec Mail Security for Exchange):

The attachment "--FILENAME--" located in message with subject "--SUBJECT--", located in --USERNAME--/Inbox has violated the following policy settings:

Scan: Auto-Protect
Rule: Unscannable File Rule
The following actions were taken on it:
The attachment "--FILENAME--" was Quarantined for the following reason(s):
Timeout occurred while scanning this item

Event ID 9791 (MSExchangeIS Mailbox Store / Background Cleanup):

Cleanup of the DeliveredTo table for database 'First Storage Group\Mailbox Store (--SERVERNAME--)' was pre-empted because the database engine's version store was growing too large. 0 entries were purged.
 
The store finally crashed when this event log error was recorded:
 
Event ID 623 (ESE / Transaction Manager):
Information Store (10180) First Storage Group: The version store for this instance (0) has reached its maximum size of 155Mb. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.

Possible long-running transaction:
SessionId: 0x7FD31E20
Session-context: 0x00000000
Session-context ThreadId: 0x00000EB0
Cleanup: 1

Here is what happened:

One of our users was sent a few 200MB-ish

Rebooting the server fixed the problem temporarily, but within a few minutes we began receiving the same event log errors (minus the ESE error) again, leading me to believe that the store was on its way to another spectacular crash.

In order to remove these messages that were causing us so much grief, I used a utility called MFCMapi.exe.  It is extremely useful for multiple back-end Exchange tasks, and can be downloaded here.  You have to set up an Outlook profile for an Exchange Administrator on your PC, then launch MFCMapi.exe:

• Click Session --> Logon only (Does not display stores).
• Select the profile of the Exchange Administrator and click OK.  Nothing will change on the screen; that's okay.
• Click MDB --> Get Mailbox Table...
• Type in the username/password of the Exchange Administrator you created the outlook profile for, if required.
• Leave the defaults in the next window and click OK.
• This will pull up a list of all the mailboxes on the server you set up the Outlook profile to connect to.
• Find and double-click the problem user (identified in my case by the Symantec Mail Security for Exchange event log errors).
• Expand Root Container --> Top of Information Store --> Double-click the folder that contains the problem email.
• You can sort the contents in the window that appears by size.  This is the easiest way to find large emails.  Once I found the 200Mb email that was causing the problem I right clicked it and selected Delete Attachments...
• Leave the pop-up blank to delete all attachments in the selected email.

After taking the steps listed above I didn't have any further issues.  I am still not sure how a 200Mb email ended up in our mail store, however...  Let me know if you have any ideas!

Single Outlook client can't connect; Exchange errors!

A few days ago I encountered a strange Exchange / Outlook problem.  One of our Outlook 2003 installations couldn't connect to our Exchange 2007 server, no matter what we tried.  The user's PC was able to ping the server, and even connect via web-mail, but no Outlook!  The user was getting all sorts of connectivity errors in the Outlook client itself, primarily:

Unable to open your default e-mail folders. The Microsoft Exchange Server computer is not available. Either there are network problems or the Microsoft Exchange Server is down for maintenance.

After hours of troubleshooting, I noticed that a specific event log error kept popping up on the Exchange server when we would try to connect:

ComputerName= Logfile=Application Type=Error SourceName=MSExchangeIS Category=6 CategoryString=General EventCode=9646 EventID=-1073338962 TimeGenerated=20100426134708.000000-420 TimeWritten=20100426134708.000000-420 Message=Mapi session "e49d45e3-baf2-48f6-ba0c-87b944a75f88" exceeded the maximum of 32 objects of type "session".

It turns out that the Outlook 2003 client previously opened a multiple sessions with the server, hit the 32 session limit, then crashed or otherwise left the connections hanging open, and this was preventing all new connection attempts from succeeding.

Restarting the Exchange services would have fixed this problem, but since this would have disconnected hundreds of clients (this was during the day), restarting services wasn't an option.

It turns out that the hung connections can be closed by the Windows Sysinternals tool called TCPView (LINK).  Here is how to close the connections:

1. Open the Exchange Management Shell (I'm assuming you're running Exchange 2007), and run this command (USERNAME should be the user attempting to connect via Outlook):
• Get-LogonStatistics USERNAME | Format-Table username,logontime,clientipaddress
2. This will output all 32 connections.  Make a note of the IP address(es) that the hung connections were initiated from.
3. Download TCPView (link above) and run it on the Exchange server to which the Outlook client is failing to connect.  If you have a busy server it may take a few minutes for TCPView to gather all the info.  You can turn off the refresh by selecting View --> Update Speed --> Paused.
4. Click Options --> uncheck Resolve Addresses.
5. Sort first by Remote Address, then by Process, and type "ST" in the window to scroll down to the "store.exe" process connections.
6. Find the connections to the store.exe process from the IP address(es) you noted in step 2 above.  Select all of those connections, right click, and click Close Connection.
7. Run the command in step 1 above again in the Exchange Management Shell to verify that the user has no more open connections.
8. The Outlook client should now be able to re-connect to the Exchange server successfully.  You may run into this issue again if you do not resolve whatever was causing the excess connections to begin with.

Integrating VMware VDR with Backup Exec

VMware VDR is working properly for our on-site backups and data recovery (others have had a lot of problems with it, but it works for us), and I needed to figure out a simple way to get our backups off disks and onto tape for off-site storage.  Although is possible to replicate the VDR backups for off-site storage if you are using a SAN with replication technology, I can’t allocate enough space on our SAN right now to store/replicate the backups, so that is not an option.

Our VDR appliance is sending backups to a simple NTFS share on our backup server.  The backup server has Symantec Backup Exec (2010) and an attached LTO4 tape library that I will use copy the VDR backups to tape.  The task seems pretty straightforward, except for the fact that the VDR appliance needs to be powered off in order to ensure valid backups.  You don't want VDR to start an integrity check or backup job in the middle of your copy to tape.

The first thing I thought of was to configure a scheduled task in vCenter to shut down the VDR appliance in the morning and turn it back on at night (presumably after the tape backup has completed).  This would technically work, but I don't like having VDR powered off all day.  It's a pain to power it on each time you want to check settings or to let the integrity check run. 

The solution I came up with was to add a pre-command and a post-command to the Backup Exec job to power off and power on (respectively) the VDR appliance.  The first thing I had to do was install the vSphere CLI on the backup server (I installed it to C:\vCLI, for ease of access).  Once this is installed, you'll need to create batch files for the pre/post commands (I'm not sure why, but it doesn't work if you put the commands directly into Backup Exec), then add the proper pre/post commands into Backup Exec after verifying that they do what you want (shut down and restart the VDR appliance):

1. Install the vSphere CLI on the server that runs Backup Exec.
2. Run the following command from the vSphere CLI:
• C:\vCLI\bin>vmware-cmd.pl -H ESXHOST -U USERNAME -l
3. Where ESXHOST is the ESX/ESXi host that the VDR appliance is running on and USERNAME is the account that has access to this ESX server (usually "root").
4. The previous command will give you a list of vmx files.  Copy the line that lists the VDR appliance's vmx file.  It will probably look something like this:
• /vmfs/volumes/4bbb70ad-10e222d0-4d79-001018645db2/VDR/VDR.vmx
5. This gives you all the info you need to create your pre/post batch files.  They will be (starting with the pre-command batch file):
• C:\vCLI\bin\vmware-cmd.pl -H ESXHOST -U USERNAME -P PASSWORD "VMX-FILE-PATH" stop soft
• C:\vCLI\bin\vmware-cmd.pl -H ESXHOST -U USERNAME -P PASSWORD "VMX-FILE-PATH" start
6. Test these batch files before starting your backup job, and make sure that they work properly!  The first one should gracefully shut down your VDR before the backup job begins, and the second one will power it back on when the backup completes.
7. Add the batch files you created to the pre/post command fields inside your Backup Exec job.
8. Now theVDR appliance will only be offline during the actual backup process!