Cisco’s Fabric Path

I spent part of yesterday parsing Cisco’s data center announcements here and here including Fabric Path.  I was trying to rationalize facts from media coverage.  In trying to sort out the facts, I did the following:

1. I read a few sell side reports on the Cisco’s announcements, but there was not as many as I had expected.

2. I read numerous blogs including Cisco’s blogs.

3. I read most of the software release notes and data sheets.

4. I also configured and priced out models using the Nexus 7009 / 7018 / 5548 and Fabric Path.

My conclusion is there are some hardware improvements in terms of port densities, which help with cost per port, but this is just a lot of marketing.  I should support this statement with some facts because the majority of the reports I read seem to be written by people who have very little, if any knowledge about the company and technology they are covering.  I think the best way to understand technology solutions is to build and price the solution.  Never believe what the marketing people tell you because they are like a car dealer who gives you a nice price, but forgets to tell that air conditioning, power package, entertainment systems and heated seats are extra.  Marketing people tend to do their job, which is to find all sorts of ways to promote their products that are generally meaningless to the core function and value of the product.

I built several pricing models for the Nexus line using reseller pricing in the market yesterday.  I built a set of complex models in which I make margin assumptions for Cisco and their VARs, but I think these models are too complex for a blog.  I condensed the models to show list and street pricing for three components in four configurations (I did more configurations with the complex models, but for the blog I am going with simplicity).

There are two important numbers in the following chart. The first is the cost of the 10GbE spine ports and the second is the cost of the 10GbE client ports (ToRs) which include the spine cost.  I chose a 3:1 subscription model, which means there are three (3) client 10GbE ports for each 10GbE uplink port to the spine which is either a Nexus 7009 or Nexus 7018.  I show the spine and client port costs at list and street and with and without optics (SFPs).  For simplicity I used short reach 850NM SFPs priced as new, not refurbished.

In Cisco’s announcement there was a new fabric model (Fab-2) for the Nexus 7k series and a new 48 port 10GbE module.  This helps with cost per port calculations.  They also announced Fabric Path, which is a software enhancement that needs to be purchased per N7K.  The release notes for the N5K say that Fabric Path will be supported in a future software release and I do not know if this will be included in the base software so I left it blank.  CSCO also announced a stackable 16x40G switch for the 3000 series.

What this chart shows you is the port costs in a N7009 (spine) and N5548 (ToR) arrangement using 3:1 subscription using the old hardware.  The second column is the same configuration, but using the new Fabric-2 module, 48x10GbE card and Fabric Path.  The third column is a N7018 (spine) and N5548 (ToR) configuration.  The <$1,200 per port list price called out by the Cisco marketing slide is calculated here at $1,146.  The final column uses the N7018 (spine) and N5548 (ToR), but I priced in the LISP software which is required for inter-data center scale.  According to the Cisco release notes for NX-OS 5.2 “The Locator/ID Separation Protocol (LISP) is a new routing architecture designed for Internet scale and global reach across organizations. Cisco NX-OS Release 5.2(1) introduces LISP VM mobility which is designed to enable global IP endpoint mobility across private networks and the Internet.  LISP functionality requires the use of the 32-port 10-Gigabit Ethernet SFP+ I/O module (N7K-M132XP-12) or the 32-port 10-Gigabit Ethernet SFP+ I/O module XL (N7K-M132XP-12L). These modules can be used independently or combined with F1 series modules in proxy mode to deliver LISP functionality in a Cisco Nexus 7000 Series switch. Traffic received on other M-series modules will not be processed by LISP because they cannot operate in proxy mode.  LISP does not require a new license. It can be enabled with the Transport Services Package license (N7K-TRS1K9).”

If you are a shareholder of CSCO, you are very happy with the Fabric Path announcement and the new Nexus hardware.  CSCO is attacking Juniper’s QFabric scale with all sorts of numbers, but rest assured there is very little reduction in price and profit margin.  This is good news is, customers staying on the CSCO path will continue to purchase vast amounts of hardware to scale the network vertically.  The other action by CSCO was to fend off the ankle biters like Arista and that is exactly who the new 3016 switch is targeted against.  Pricing is not available in the channel for this product as of yesterday.

When I wrote a post on how Cisco lost their way here and part 2 here, one aspect that did not occur to me at the time was how many isolated platforms are inside the company.  When I read this post by a Google engineer who accidently posted his rant on what is wrong with Google on his Google+ account, it reminded me of how things must look inside Cisco which is why we got the announcements yesterday.


* It is all about the network stupid, because it is all about compute. *

** Comments are always welcome in the comments section or in private.  Just hover over the Gravatar image and click for email. ** 

One thought on “Cisco’s Fabric Path

  1. Very good blog, as usual, you are looking at cost models which are interesting the more interesting thing to me is how will these products actually work. Look a the per port buffers available on the new F2 line card, it is being touted as an aggregator of many ToR switches. The issue is the buffering based on the data sheet the F2 line card has ~6MB per 4x10GbE ports. The interesting part of this is looking at how, with a VoQ architecture, the N7018 is actually going to work in a live network. Here is the math:

    Assumptions for the calculations:
    – Each group of 4x10GbE has 6MB (based on data sheet)
    – 768 x 10GbE is the desired configuration

    6,000,000B (size of buffer) / 768 (# of ports) = 7,812.5B (the max packet size that can be buffered)

    This does not factor in queuing for broadcast and mutlicast traffic. Something does not add up as nobody would build a line card that can not switch a jumbo frame would they?

    It could be explained in a couple of ways:

    1) The data sheet is wrong and they meant a larger number of the per SoC buffering
    2) There is some VoQ sharing which would mean there are other issues with HoL blocking and non-line rate performance

    Even if they worked out some magic with VoQs there is such a small amount of buffers for a box that is suggested can aggregate 10K+ servers there is going to be massive packet loss – not that Ethernet is perfect and it does drop packets but in this case things are going to break. I can get into how this is going to really break PFC/DCBX lossless applications but that is a longer post. Anyway great articles just wanted to put something else on the table – the size of the buffers means this is a disaster waiting for a couple of customer to deploy it and realize their network has….

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.