Top-of-Rack Switching - a Power Saver?

By David Gross

In just about any survey about energy efficiency, data center managers reply that it's their number one concern, a major priority, or give some other kind of indication that they are highly focused on saving energy.   Then they go out and buy power hungry, high density GigE line cards for their switches and routers.

Many technology surveys ignore the fact that buyers often say one thing and do another, but investors can't.   And when it comes to network equipment, the best ways to reduce watts per Gbps include buying an OC-768 line card, which costs over half a million dollars, or an InfiniBand switch, which typically requires a Clos topology, which is still uncommon outside of supercomputing.

As an example, Voltaire's 4036E offers 1.36 Tbps of InfiniBand switching with just 240 Watts, or just .18 Watts per Gbps.   Meanwhile, the 32-port GigE/1 port 10GigE uplink card for the Nexus 7000 consumes 385 Watts, or 9.17 Watts per Gbps, 50x more, and there are far more Nexus switches than Voltaire devices in most data centers.

There is often a trade-off between power consumption and features, because much of a high-speed router or switch's power is tied to TCAMs, packet processing, and the memory/forwarding requirements of large routing tables.   However, with few of these sophisticated features, Top-of-Rack switches offer tremendous bandwidth with limited power.  Force10's S4810 ToR switch, for example, requires just .44 Watts per Gigabit.   But its larger Exascale switch has a 10-port 10 GigE line card that needs over 3 Watts per Gig.    Configuration flexibility has a price.

While many data center managers are still resisting ToR switches, there is roughly a 40% Watt per Gbps improvement just from upgrading from GigE to 10GigE.    And while there is a significant cost penalty at 10GigE going from short reach 850nm optics to longer reach 1310nm and 1550nm optics, there is no power penalty.    Here again, the least capital efficient networking options and the most power efficient.  

While the IRR on buying more expensive ports to save power is often negative, there are at least a couple options developing to reduce Watts/Gbps that don't require an entirely new topology, or a multi-million dollar router.