Build a fast, reliable, and affordable switch infrastructure for your datacenter

by George Ou | Jan 11, 2007 8:00:00 AM

Tags: Network technology, data center, George Ou, rack, Cisco Systems Inc.

Takeaway: The traditional approach to setting up a high-speed network switch architecture is expensive and time-consuming--and needlessly complex. George Ou explains where the conventional model falls short and offers better alternatives.

This article and a companion Visio diagram are available as a download.

Supplying high-speed data interconnects for the datacenter has always been a complex and costly project. The conventional way to arrange a row of server racks wastes time and money, while resulting in more complexity and less reliability. Using that approach, you either buy a centralized chassis solution with a 32-gigabit or higher backplane (which costs as much as a house) or settle for mediocre performance using smaller distributed switches using 1-gigabit fiber or copper interconnects. Switching to 10-gigabit fiber or copper interconnects will quickly jack up the price, and it still leaves you too many individual switches to manage.

The following series of network switch architectures will explain the downsides of the conventional approach and show you a better solution. You can download the Visio diagram here to see them up close.

The centralized model in Figure A is fast and simple to manage, but it means paying a fortune for the switch and paying a contractor to run patch panels from the central rack to the six surrounding racks. You're also wasting an entire rack just to house the panels, which is why there are seven racks in Figure A (compared to only six racks in our other two models).

Figure A

A switch like the Cisco 6509 costs anywhere from $100,000 to $300,000 depending on the options you choose. You will also need to buy 24 48-port patch panels, which will probably cost another $5,000, plus a day's worth of contractor labor to punch down all the wiring. This also adds complexity to the cabling if a link should ever fail and you'll need to spend time to troubleshoot where the break in the line is. It could be in the central cabinet or the remote cabinets or it could be a faulty line in the punch downs or the panel interconnects.

It's also common to stack cheaper switches like the Cisco 3750 in place of the Cisco 6509, but you still need to run the patch panels and lose a whole rack to network infrastructure.

The decentralized model in Figure B is typical of setups that are on a tight budget. You can use slightly cheaper switches, like a Cisco WS-C2950G-48 Layer 2 switch, which are in the $2,000 range, including the two gigabit GBICs. But you'll still need one Layer 3 switch, like the $8,000 Cisco 3750G-24TS, to act as the routable core.

Figure B

The problem is that once you use up the two gigabit ports just to link up to the 3750, you're all out of gigabit ports and you're left with 48 10/100 FastEthernet ports. None of your servers will be able to connect at gigabit speeds even if they need it, like the file or backup server, unless you run a cable across the racks to the 3750. I've also known people to use the more expensive Cisco 3750 for all the switches in this configuration. That's a terrible waste of resources, which I'll explain in the next model. Furthermore, you'll have to manage every switch in the picture individually, which is a management nightmare. If you need to change the configuration or upgrade software images, you'll have to do it one at a time.

Figure C shows how a series of higher-end Cisco WS-C3750-48TS Layer 3 switches can be distributed across up to nine racks--yet all nine 3750s are managed as a single logical switch. You can SSH into a single switch, and you'll be able to manage all nine switches or upgrade the software on all of them at once.

Figure C

The thick 1- or 3-meter stacking cables form a loop connecting all the 3750s. They carry 32 gigabits per second and won't use up any SFP slots on the 3750. This leaves you with four open gigabit ports per rack, which can be fiber or copper. (Fiber is used for cable runs that are more than 100 meters, although that's obviously not needed here.) This means you have four gigabit ports open per rack that can be used to connect servers like file/backup servers that require extra bandwidth. The setup in Figure C bypasses the need for patch panels because the servers in each rack go straight up to the switch. Here are the part numbers for the stacking cables of varying lengths:

3750 stacking cable part numbers

72-2632-XX CABASY (0.5 meter cable) [This is bundled with switch]
72-2633-XX CABASY (1.0 meter cable)
72-2634-XX CABASY (3.0 meter cable)

The Cisco 3750 has built-in routing with enhanced software options that can support even OSPF and BGP. There are a variety of 3750 switches, some of which carry power over Ethernet. Some have all gigabit ports, and some models even support 10-gigabit Ethernet. The price can vary from $3,000 to $10,000 depending on the model and SFP options. The WS-C3750-48TS-S shown in Figure C is roughly $4,100 for the lowest reputable street price. If one of the switches in the stack fails, the looped topology will still allow the remaining eight switches to remain active while you replace the ninth switch. Once the switch is replaced/repaired, you can hot-plug it back into the stack.

The bottom line

We can choose from three types of switch architecture to light up a data center. Here's how it all breaks down:

Figure A will cost you roughly $200K. You will get 288 10/100 FastEthernet ports, along with a couple of gigabit ports on the two SUP cards. If you stack a bunch of 3750s in place of the 6509, it will cost a lot less money, but you'll still pay more for the complex patch panel arrangement and make cabling more complex.

Figure B will cost you roughly $20K. You will get 288 10/100 FastEthernet ports and 16 gigabit ports open on the 3750. Note that if the one 3750 dies, the whole network is down. You won't save that much money in this configuration, and there is a single point of failure if the 3750 in the middle dies. You could buy two 3750s for redundancy, but that would jack up the costs another $8,000--at which point, you might as well go with Figure C.

Figure C will cost you roughly $25K. You will get 288 10/100 FastEthernet ports, along with 24 open gigabit ports evenly distributed among all six racks. If you have 12 racks, you can simply duplicate this configuration. Since you can scale up to nine stacked switches, you can have rows of nine racks fed in this configuration. Some 3750 models have 10 gigabit ports, which can be used to interconnect many rows of nine racks. Any failure in any one of the switches will cost you only 52 connections in a single rack but not affect any of the other racks. This buys you the same amount of redundancy as using a single 6509 in Figure A if one of the 6509 blades fails. But you can always buy two independent sets of everything and use dual Ethernet servers if redundancy is that much of a priority. At this price, you can afford to double up on everything and still spend only $50K--which is far less than a single 6509--and have better survivability.

4 comment(s)