When to Upgrade to 40 Gigabit?

By David Gross

ZDNet has a good article out on the decision to upgrade to 40G.   But for all the talk and excitement over the new standard, the reality is that line rate is not as important a measure of network capacity as it used to be, especially with multi-gigabit port prices having more to do with transceiver reach, cabling options, and port densities than framing protocol.    10GBASE-SR, the 850nm 10 Gigabit Ethernet multimode standard, for example, has a lot more in common with 40GBASE-SR4 than it does 10GBASE-EW, the 1550nm singlemode, WAN PHY, standard.   Moreover, a 40 Gigabit port can can cost as little as $350, if it's configured on a high-density Voltaire InfiniBand switch, but over $600,000 if it's configured to run Packet-over-SONET on a Cisco CRS.

Within the data center, which line rate you choose is increasingly falling back in importance to which transceiver you choose, what type of cabling, whether to aggregate at end-of-row or top-of-rack, and so forth.   Moreover, as I wrote a few weeks ago, the most power efficient data center network is often not the most capital efficient.    So rather than considering when 40 Gigabit Ethernet upgrades will occur,  I think it's more important to monitor what's happening with average link lengths, the ratio of installed singlemode/multimode/copper ports, cross-connect densities in public data centers, the rate of transition to IPv6 peering, which can require power-hungry TCAMs within core routers, and especially whether price ratios among 850, 1310, and 1550 nanometer transceivers are growing or shrinking.   So rather than wondering when 40G will achieve a 3x price ratio to 10G, it's equally important to consider whether 10G, 1550nm transceivers will ever fall below 10x the price of 10G, 850nm transceivers.

Line rate used to be far more important within this discussion.   When 802.3z (the optical GigE standard) came out in 1998, it wasn't that it was 3x the price of Fast Ethernet that mattered to data centers, but that it was cheaper than 100 Meg FDDI, which was the leading networking standard for the first public web hosting centers.   The wholesale and rapid replacement of FDDI with GigE was largely a result of line rate - more bits for less money  - and an economic result of the low price of high volume Ethernet framers.    But over a gigabit, production volume of framers gives way to the technical challenge of clocking with sub-nanonsecond bit intervals, and silicon vendors have had the same challenges with signaling and jitter with Ethernet that they've had with InfiniBand and Fibre Channel.   This is a major issue economically, because widely-used LVDS on-chip signaling is not Ethernet-specific, and therefore does not allow Ethernet to create the same price/performance gains over other framing protocols it did at a gigabit and below. 

Another factor to look at is that all the 40 Gigabit protocols showing any signs of hope within the data center  run at serial rates of 10 Gigabit, whether InfiniBand or Ethernet-framed, because no one has come up with a way to run more than 10 Gigabit on a serial lane economically, even though it's been technically possible and commercially available for years on OC-768 Packet-over-SONET line cards.   In addition to the high costs of dispersion compensation on longer reach optical links, transmitting a bit less than 1/10th of a nanosecond after the last bit was sent has proven to be a major economic challenge for silicon developers, and likely will be for years.

So as we look beyond 10 Gigabit in the data center, we also need to advance the public discussion beyond the very late-90s/early 2000s emphasis on line rates, and look further into how other factors are now playing a large role in achieving the best cost per bit in data center networks.

Force10 Announces 40 Gigabit Ethernet Top-of-Rack Switch and Line Card

By David Gross

Extreme and BLADE Networks announced the launch of their 40 GigE ToR switches a few weeks ago, and now you can add Force10 to the list of vendors selling the devices.    The company announced a new Top-of-Rack switch today, the S4810, which features 4 40 GigE uplinks, and will be available for sale by year-end.   But Force10 didn't stop with the ToR switch news, it also announced a 40 GigE line card for its Exascale core switch.

IBM currently re-sells Force10's S60 ToR device, but the company did not say if the S4810 would be included in the IBM resale program.   It did say that it would offer lower power consumption than BLADE's RackSwitch, which was an important statement, because it means that unlike ToR switches with Voltaire, Juniper, and other company's logos on them, the S4810 is NOT an OEM of the BLADE RackSwitch.

Unlike Extreme and BLADE, Force10 did not publicly announce pricing for its 40G ports, but the other vendors are charging around $1,000 per port on their ToR switches.   But these are for short-reach links.   Longer reach 40GBASE-LR4 ports will be much more expensive when they come to market, and 40 Gigabit OC-768 ports on routers cost half a million dollars, or 500x a 40GBASE-CR4 switch port.   Meanwhile, 40 Gigabit QDR InfiniBand ports now sell for under $300, with similar short-reach QSFP links as the Ethernet boxes.  Ultimately, with values between $250 and $500,000, you can't really say there is such a thing as a price for a 40 Gigabit port, but rather a price based on transceiver type, port density, and most importantly, link length.

Mellanox Revenue Up 16%, Continues to Advance 40 Gigabit in Data Centers

By David Gross

Mellanox (MLNX) reported quarterly revenue of $37.8 million Wednesday night, up 16% year-over-year, but down 5% sequentially, and slightly ahead of guidance of $37-$37.5 million.

The stock lost about a third of its value the day after its second quarter call in July, when it announced revenue would drop sequentially.   Since then, the stock has rebounded 30%, making up most of the July losses.

The company has had a very strong balance sheet since going public in 2007, and it finished the quarter with $248 million of cash, up $38 million since the end of 2009, and with no debt.

Mellanox has been experiencing a bit of an awkward shift to mixing InfiniBand and Ethernet products, after having long been the leader in InfiniBand silicon.    Having gotten a significant share of its revenue from 40 Gigabit QDR InfiniBand for the last two years, it has now crossed over to 40 Gigabit Ethernet,  most notably with its ConnectX-2 EN 40G, which  was the first 40 Gigabit Ethernet Server NIC.   The company has also worked with the InfiniBand Trade Association to develop RoCE, or RDMA over Converged Ethernet, which is basically InfiniBand over Ethernet.  

Traditionally a silicon vendor, Mellanox has also been transitioning to NICs and silicon products for Landed-On-Motherboard chips.   But these shifts have gone back and forth the last couple quarters, and silicon shipments replacing higher revenue adapters is one reason why revenue was down sequentially.   While the company has not always been clear on the extent to which it plans to focus on silicon vs. cards, I increasingly think its strong position in both supercomputing clusters and the rapidly developing market for 40 Gigabit networks should help restart its growth in 2011.

Mellanox 40 Gigabit InfiniBand Switch Gets a Boost from IBM

By David Gross

There's been lots of news this past week on 40 Gigabit Ethernet Top-of-Rack switches.   Not to be left out, 40 Gigabit InfiniBand is also making news, with IBM (IBM) choosing Mellanox's (MLNX) IS5000 InfiniBand switch silicon for its iDataPlex and Intelligent Cluster platforms.

iDataPlex is often used in HPC environments, and this announcement is a strong endorsement for advancing 40 Gigabit InfiniBand in supercomputing clusters.   The platform incorporates both GPUs and CPUs, and IBM previously added QLogic's (QLGC) 12000-series of 40 Gigabit QDR InfiniBand switches to its Intelligent Cluster package. 

While list pricing for the 40 Gigabit Ethernet ToR switches is between $1,000 and $1,500 per port, street pricing for QLogic's 40 Gigabit InfiniBand switch is less than $300 per port.    While both the Ethernet and InfiniBand use QSFP transceivers, the price difference is likely the result of density - with the 12000-series supporting 2.88 Terabits per second send and receive, and 36 40G ports, compared to 1.2 Tbps and a maximum of four 40G ports on BLADE's recently announced RackSwitch G8264.    Additionally, the 12000 series comes as is, with very few configuration options.   However, with a little more production volume, the Ethernet ports should begin to creep down into the three figures as well.   I would also expect the applications for each to remain different, with the ToR switches serving the enterprise data center market, and the InfiniBand switches primarily going into supercomputing clusters.   In either case, we're still talking about very short-reach links, 40 Gigabit links in telco networks still cost over 1000x as much.

IBM to Sell Force10's Top-of-Rack Switch

By David Gross

Lots of interesting things happening in the Top-of-Rack (ToR) switch market.   Last week, Extreme Networks (EXTR) showcased a demo of it ToR switch running a 40 Gigabit Ethernet link in a supercomputing network, while BLADE Network Technologies, soon to be part of IBM (IBM), announced a new ToR product which offers 40 Gigabit Ethernet ports.   Not to be forgotten, Force10 Networks announced that it will be selling its S60 ToR switch through IBM. 

Force10 has a well-established relationship with IBM, centered around its high-end E series switches that are used in data center cores and supercomputing clusters.   While some core router/switch vendors, notably Juniper (JNPR), have entered the ToR market by OEM'ing other suppliers' products, the S60 appears to be home grown, and runs Force10's FTOS software.  

Like other ToR switches, the S60 is designed around price/performance, not a wide selection of configuration options.   10 Gigabit Ethernet uplinks are available as an option, using SFP+ transceivers, and LAN PHY framing, which is essential not just for short reach but to keep costs down, as 10G WAN PHY ports remain more expensive than the 40 Gigabit Ethernet ports on BLADE's and Extreme Networks' ToR switches.   The reason the S60 does not feature 40 Gigabit ports is that it is more of an entry-level product than the others, with a send and receive throughout capacity of 136 Gbps, compared to 1.2 Tbps (Terabits) for the BLADE box.    And while ToR switches are not overloaded with software features, all three vendors support IPv6.

40 Gigabit Ethernet Demonstrated on Supercomputing Network

By David Gross

Earlier this week, BLADE Network Technologies released its RackSwitch G8264, one of the first Top-of-Rack switches to feature 40 Gigabit Ethernet ports.    Extreme Networks (EXTR) has also added 40 Gigabit capabilities to its Summit X650 ToR box, which was used by two European research labs, CERN and SARA, to demonstrate a 40 Gigabit Ethernet link between their facilities in Geneva and Amsterdam.

As part of the demonstration, an Extreme-provisioned 40GigE link was fed to a long-haul optical transport platform in Geneva, which then passed the traffic back to another Extreme switch in Amsterdam.   The switches were used to display a computer-generated video of the Big Bang, and were part of the 10th Annual Global Lambda Grid Workshop, where research lab engineers gathered to discuss the latest advancements in high-speed networking.

Extreme lists pricing for the 40 GigE ports on the Summit X650 at $1,000, while BLADE lists theirs at $1,350.  However, these are for short-reach links.   Pricing on longer-range 40GBASE-LR4 ports, which will transmit up to 10km without a push from any optical transport devices, will likely cost well into the five figures.

BLADE Introduces 40 Gigabit Ethernet Top-of-Rack Switch

By David Gross

BLADE Network Technologies, soon to be part of IBM, has released a ToR switch, the RackSwitch G8264, that can be configured with four 40 GigE ports, and offers up to 1.28 Terabits of throughput send and receive combined.  

The switch can be configured with as many as 64 10GigE ports, using SFP+ transceivers.    The 10G ports have an amortized cost of $350 each, although I'm sure anyone using the long-reach transceiver option will be paying a little more.   Pricing for 10GBASE-LR SFP+ modules has been running about 2x the cost of 10GBASE-SR SFP+ counterparts.   The 40G ports (only 40GBASE-CR4 is available) cost $1,350 each, and can be configured with QSFP Direct Attach Cables running 1-3 meters.   This is cheaper than most 10GBASE-LR ports, and significantly cheaper than 10 Gigabit Ethernet WAN PHY ports, which still go for well over $10,000.  In multi-gigabit networks, typically it's link length, not line rate, that determines price.   

40GBASE-CR4 is a 4x10 standard, and BLADE allows the 40G ports to be broken down into four separate 10G ports for an all-10G configuration.   There is still plenty of work to be done on serial transmissions over 10G, and it's not clear serial rates over 10G will ever be cost effective in the data center, or anywhere else for that matter.  Nonetheless, 40 Gigabit Ethernet has finally moved from analyst/standards committee/PowerPoint concept to data center network reality.

UPDATE: in the press release BLADE did not mention whose silicon they were using to create the "first terabit switch with a single ASIC", but I believe it's likely privately-held Fulcrum Micro, whose silicon sits in BLADE's RackSwitch G8100

Common Electrical Interface for 25/28G - Possibility or Pipe Dream?

by Lisa Huff

Monday, I sat through a workshop hosted by the Optical Interconnecting Forum (OIF) on its “Common Electrical Interface (CEI) project for 28G Very Short Reach (VSR).” What quickly became clear to me was that I was in a room of VERY optimistic engineers.

I sat through presentations that were characterized as “Needs of the Industry,” which consisted of content from the leaders of IEEE 802.3ba (40/100G Ethernet), T11.2 (Fibre Channel) and InfiniBand standards groups. Yet all of these representatives made sure they carefully stated that what they were presenting was their own opinions and not that of their standards groups, which I found odd since most of what they showed was directly from the standards. Legalities I guess. I also noticed that they never really sited any kind of independent market research or analysis of what the “needs of the industry” were. For instance, one speaker said that InfiniBand has a need for 26AWG, 3-meter copper cable assemblies for 4x25G operation in order to keep the cost down within a cabinet. Yet, he did not present any data or even mention what customers are asking for this. Maybe this exists, but to me unless it is presented, the case for it is weak. I do have evidence directly from some clustering folks that they are moving away from copper in favor of fiber for many reasons – lower power consumption, weight of cabling, higher density, and room in cabinets, pathways and spaces.

Today, data center managers are really still just starting to realize the benefits of deploying 10G, which has yet to reach its market potential. I understand that standards groups must work on future data rates ahead of broad market demands, but this seems extremely premature. None of the current implementations for 40/100G that use 10G electrical signaling have even been deployed yet (except for maybe a few InfiniBand ones). And, from what I understand from at least one chip manufacturer who sells a lot of 10G repeaters to OEMs for their backplanes, it is difficult enough to push 10G across a backplane or PCB. Why wouldn’t the backplane and PCB experts solve this issue that is here today before they move onto trying to solve a “problem” that doesn’t even exist yet?

Maybe they need to revisit optical backplanes for 25G? It seems to me that 25G really won't be needed any time soon and that their time would be better spent on developing something that would appear to have relevancy beyond 25G. To me, designing some exotic DSP chip that would allow 25G signals to be transmitted over four-to-12 inches of PCB and maybe 3m of copper cable for one generation of equipment may not be very productive. Maybe this is simpler than I anticipate, but then again, there was a similar but a little more complicated problem with 10GBASE-T and look how that turned out...

Network Utilization Has Nothing to do with Upgrading Bandwidth

by David Gross

One of the questions I've been asked by hedge funds is whether networks are full enough to justify upgrades to 40 or 100 gigabit, as if communications networks are like airplanes or railcars, and only need to be expanded if they're already full. But this misses the point of why upgrades beyond 10 gigabit are needed, especially in the data center, where it's long been possible to provision 10 gigabit links for much less cost than longer reach 1 gigabit connections in telecom networks.

Financial professionals more than anyone should understand the upgrade past 10 gigabit is primarily about latency, not bandwidth. And one of the primary contributors to latency, serialization delay, drops proportionally to increases in line rate. A 1500 byte packet, for example, has a 1.2 microsecond delay at 10 Gbps, but a .3 microsecond, or 300 nanosecond, delay at 40 Gbps. With financial services, one of the leading buyers of high capacity switches, the upgrade is therefore cost justified by the value of reducing execution times by 900 nanoseconds, which is far more than the cost of a new switch for many firms. None of this has anything to do with whether the existing network peaks at 10% utilization during the day, or at 60%.

Even in longer-reach networks, upgrades are tied to revenue, not just network utilization. When AT&T (T) upgraded its MPLS core to 40G, it included nine out-of-region markets where it did not offer U-Verse or any video service. The financial justification was to maintain low latency for commercial customers whose traffic was riding that MPLS networks, many of whom spend tens of thousands a month with the carrier.

Bandwidth is often described as a faster link or fatter pipe, the latter might make sense when talking about consumer video, but no one's getting 40 or 100 gigabits to their home anytime soon. In data centers and even long-reach corporate networks, it makes more sense to think of upgrades in terms of raw speed, because a faster path, not a wider lane, is essential when you're trying to make a quick trade.

ICO, not TCO

One of the most overused sales metrics in technology is “TCO”, or total cost of ownership. For decades now, sales teams have been abusing financial principles in order to get their business customers to buy hardware or software products using this made up metric. There are countless TCO models flying around Silicon Valley, and their ability to move products has more to do with sales skill than financial reality.

But in investigating products that have lasted far longer than the hype accompanying their release, one thing I've found consistently is that they don't have a low TCO, but a low incremental cost of ownership, or what I'll just call ICO, because like Washington, Silicon Valley likes three letter acronyms.

So what is ICO exactly?

Now calculating TCO often involves beating up Excel, plugging data points into macros, and ensuring that in the process, spreadsheet inputs lose all connection to operating reality. Incremental cost of ownership is a highly technical calculation that looks something like this:

Total amount of money needed to purchase product + 0 = Incremental Cost of Ownership

In looking at products and protocols that have been successful over the years, one thing stands out. Not their TCO, but their ICO. From Cisco's 6500 series switches to Gigabit Ethernet, it is hard to find a widely deployed technology that does not have a reasonable incremental cost of ownership. It's important to note that this does not mean the products sell for low margins, Cisco's gross margins have been in the 60s and its net margins in the teens for years, it's hardly scraping by. The same is true for many of its competitors and suppliers. But in spite of all its industry strength, Cisco could do little to save technologies like ATM and RPR, which promised lower TCO though collapsing network layers, but required buying switches with very high incremental costs.

If it weren't for low incremental costs, there would not be much to say about data center networks. The reason linking up is so attractive in data centers instead of through traditional enterprise or carrier networks is because short-reach hardware components are exponentially cheaper. 40 Gig on a sub-20 meter InfiniBand port will only set you back $400, a quarter of the bandwidth on a 10 kilometer metro link will costs about 30 times as much per port. No clever model or PowerPoint will change this.

Just as IRR accompanies ROI in most well-thought out business cases, ICO should accompany any TCO presentation. Regardless of overall economic conditions, products that have promised future opex savings have typically lost out to those bringing high returns on invested capital today. By presenting both ICO and TCO, vendors would show customers that they are addressing the financial risks they deploying networking hardware, rather than just relying on promises of future opex savings.

ADC: We're Ready for 40/100 Gigabit

ADC (ADCT) issued a press release stating its support of the recently ratified IEEE 802.3ba standard for 40 and 100 Gigabit networks. In addition to fiber and copper cables, the vendor supplies optical distribution frames to data centers.

40 Gigabit InfiniBand is already selling, but many of those links are for short-reach copper interfaces. On the Ethernet side, many data centers are still upgrading from Gigabit to 10 Gigabit, so the standardization is more of a technical milestone than a major business development.