Google, Amazon, Microsoft, and Facebook buy more networking hardware than practically anyone else on earth. After all, these are the giants of the internet. But at the same time, they're buying less and less gear from Cisco, HP, Juniper, and the rest of the world's largest networking vendors. It's an irony that could lead to a major shift in the worldwide hardware market.
Over the past few years, the giants of the web have changed the way they purchase tens of thousands of the network switches inside the massive data centers driving their online services, quietly moving away from U.S.-based sellers to buy cheaper gear in bulk straight from China and Taiwan. According to J.R. Rivers -- an ex-Google engineer -- Google has built its own gear in tandem with varous Asian manufacturers for several years, and according to James Liao -- who spent two years selling hardware for Taiwan-based manufacturer Quanta -- Facebook, Amazon, and Microsoft are purchasing at least some of their networking switches from Asian firms as well.
"My biggest customers were these big data center [companies], so I know all of them pretty well," Liao says. "They all have different ways of solving their networking problems, but they have all moved away from big networking companies like Cisco or Juniper or [the Dell-owned] Force10."
The move away from U.S. network equipment stalwarts is one of the best-kept secrets in Silicon Valley. Some web giants consider their networking hardware strategy a competitive advantage that must be hidden from rivals. Others just don't want to anger their business partners in the hardware sector by talking about the shift. But cloud computing is an arms race. The biggest web companies on earth are competing to see who can deliver their services to the most people in the shortest amount of time at the lowest cost. And the cheapest arms come straight from Asia.
J.R. Rivers is one of the arms dealers. He runs a company called Cumulus Networks that helps the giants of the web -- and other outfits -- buy their networking hardware directly from "original design manufacturers," or ODMs, in China and Taiwan. And he's worked in this world for an awfully long time. He's one of the Google engineers who secretly designed a new breed of networking switch for the company's data centers, the massive computing facilities that drive its search engine and the rest of its web services.
Rivers joined Google in October 2005, after five years as a distinguished engineer at Cisco, the company that dominated the worldwide market for networking gear. At the time, Google was still connecting its servers using standard networking switches from the likes of Cisco and Force10 Networks. But these mass-market switches just didn't suit Google's unusually large operation.
"When Google looked at their network, they need high-bandwidth connections between their servers and they wanted to be able to manage things -- at scale," Rivers says. "With the traditional enterprise networking vendors, they just couldn't get there. The cost was too high, and the systems were too closed to be manageable on a network of that size."
So Google drew up its own designs -- working alongside manufacturers in Taiwan and China -- and cut the Ciscos and the Force10s out of the equation. The Ciscos and the Force10s build their gear with many of those same manufacturers. Google removed the middlemen.
The search giant does much the same with its servers, buying custom-built machines straight from Asia rather than going through traditional sellers such as Dell and HP. Because its web services were used by such an enormous number of people, Google faced all sorts of data center problems no one else faced -- problems of power and space as well as cost and logistics. So it built all sorts of custom hardware to solve those problems.
>"They all have different ways of solving their networking problems, but they have all moved away from big networking companies like Cisco or Juniper or Force10"
Now, the other giants of the web are running into the same issues, and they too are going straight to Asia for hardware. Following closely behind are companies that run large internal server farms, including financial houses and healthcare outfits.
As J.R. Rivers serves this market with Cumulus Networks, James Liao is doing much the same thing with a second startup called Pica8, offering networking gear that comes straight from the ODMs. Pica8 is a spinoff of Liao's former employer, Quanta -- one of the companies that manufactured Google's original networking switches, according to Rivers.
According to Liao, tens of thousands of switches are already being sold by the Asian ODMs directly to the likes of Amazon, Facebook, and Microsoft. And that doesn't include the gear Google has bought over the past seven years. "This is just the beginning," Liao says, pointing out that these buyers operate the biggest data centers on earth. These companies account for only a part of the $7-billion-a-year Ethernet switch market, but as more and more outfits move their operations into the proverbial cloud, the influence of these web giants will only grow.
Liao estimates that Amazon, Microsoft, Facebook and others have bought Asian network switches spanning "millions" of network ports -- i.e., connections to servers -- and he guesses that in 2011, about 60 percent of these ports provided 10Gigabit Ethernet connections. According to Matthias Machowinski -- a directing analyst with Infonetics, a research firm that tracks the networking market -- the official market for 10Gigabit Ethernet spanned about 9 million ports in 2011.
J.R. Rivers declines to name the companies he's working with at Cumulus Networks, but he confirms that some of the big-name web outfits are already buying networking switches from ODMs in Asia. In all likelihood, these companies are also purchasing switches from other sources as well. Cisco says it has a "significant presence and mindshare" in the big-name web market, and Juniper says it has a relationship with all of the top five web players, pointing out that data center networks require more gear than just switches. But the market is on the move.
The Future of 'Web Giant 3.0'
"We are continuously exploring new infrastructure technologies that may evolve further efficiencies across our portfolio. We normally have discussions with ODMs and large and small OEMs to better understand their capabilities and evaluate their products," reads a statement sent to Wired by a Microsoft spokesperson and attributed to Dileep Bhandarkar, a distinguished engineer who oversees the data centers driving Microsoft's online services. But the statement did not specifically address the purchase of networking gear.
Amazon did not respond to a request for comment about its hardware practices, and a Google spokeswoman sent us a one-sentence statement: "We work with a variety of vendors to manufacture the equipment we use in our data centers," she said. These two companies -- particularly Google -- are rather tightlipped about their data center practices.
>"This supply chain change is nascent. But it's the most exciting thing going on in Silicon Valley right now"
Facebook declined to discuss how it purchases networking gear, but in response to the secretive approach of Amazon and Google, the company has openly discussed some of its other practices, and it has actually shared its server and data center designs with the rest of the world. It purchases its servers directly from Quanta and Wistron, another Taiwanese ODM.
Martin Casado -- the chief technology officier of a third Silicon Valley networking startup, Nicira -- confirms that the hardware market is shifting to Asia. Offering a software platform that virtualizes networking gear in much the same way that VMware virtualized servers, Nicira helps some of the big web players build their networks. The Nicira platform was designed specifically for companies along the lines of Google that want to use cheap commodity switches to physically construct their network but then do all the complex management in software.
"If you're building web giant 3.0, you can go to Quanta in Taiwan and buy crates ... of switches," he says. "This supply chain change is nascent. But it's the most exciting thing going on in Silicon Valley right now."
Google Goes to Asia
According to J.R. Rivers, Google began work on its custom-built networking switches in early 2005, before he arrived at the company. In the beginning, River says, Google worked in tandem with Quanta and other Asian ODMs, but eventually, the web giant took all the engineering work in house. Basically, Rivers says, the company wasn't happy with the work the ODMs did at the time. Google engineers would design the switches, and then they would bring the completed designs to contract manufacturers in Asia, outfits along the lines of Foxconn, the Asian company that builds Apple's iPhones and iPads.
Google has never discussed its practices publicly, but rumors have long indicated that the company built its networking switches in this way. In 2007, research analyst Andrew Schmitt noticed that certain manufacturers were producing enormous numbers of chips for 10Gigabit Ethernet switches but that the switches themselves weren't actually turning up on the market. "It didn't make sense to me why someone would be building so much of a given component if there were no customers that could use it," he says. "What I was able to determine is that Google was purchasing switch chips straight from the comment suppler."
>"It didn't make sense to me why someone would be building so much of a given component if there were no customers that could use it"
The switches Google was building typically sat at the top of a rack of servers in the data center, connecting the servers to the rest of the network. As Juniper points out, this is only part of the networking hardware used in the data center, but it's a large part.
Google, Rivers says, is a unique company. It has the wherewithal and the talent to built its own switches, but other companies may not be up to the task. With Cumulus Networks, J.R. Rivers and his partner, Nolan Leake, are trying to grease the wheels. "[The other web players] are trying to figure out what the best model is, and that's one of the reasons we started up," Rivers says. "Google is unique in its willingness to build something just because they know it can be done. Most other people see a risk/reward trade-off. We seek to minimize that risk."
Though Rivers declined to name the ODMs his company is working with, he says that these are well-known manufacturers in Taiwan and China. "We've been working for the last year on opening up a supply chain for traditional ODMs who want to sell the hardware on the open market for whoever wants to buy," he says. "For the buyers, there can be some very meaningful cost savings. Companies like Cisco and Force10 are just buying from these same ODMs and marking things up. Now, you can go directly to the people who manufacture it."
This has become possible in recent years, Rivers says, because the ODMs have slowly acquired more and more engineering talent. You can now buy commodity gear from more places. "Networking is opening up much like the transition from mainframes to RISC machines and later to x86 servers," says Rivers' partner, Nolan Leake. "We're moving towards a world where customers have more control over their destiny."
'The Arms Dealer'
Before spinning Pica8 out of Quanta, James Liao was already selling similar networking switches to the big web players. Nicira's Martin Casado refers to James Liao as "the arms dealer" in this networking revolution. "He's the conduit between the rest of the world and Quanta. He knows this space better than anyone," Casado says. "And I love him because he talks like he's part of organized crime."
From July 2009 to September 2011, Liao was the senior director at Quanta in charge of product strategy for network switching and data center products. He was based in Silicon Valley, and his job was to serve the giants of the web. He declines to go into much detail about how these companies acquire their hardware, but he's unequivocal in saying that the other big companies -- Amazon, Microsoft, and Facebook -- are now following Google's lead in going directly to Asia for their gear.
Networking switches, he says, have become a commodity. "They all use the same chips. They have to same latency. They have the same bandwidth. This is a clear signal that the hardware platform is commoditizing," he says. "You can actually find a lot of [ODM] suppliers that have the capability to manufacturer and design this kind of platform."
Like Cumulus Network, Liao's new venture, Pica8, brings this low-cost networking hardware to a much larger market. In the past, one of the problems with buying directly from the ODMs is that you had build your own software to drive your switches. But Pica8 aims to provide software for those companies that don't want to build their own. The company has open sourced an early version of this software -- known as Picos -- and it plans to open source a more extensive version of the platform next month.
"We give you the hardware and the software," Liao says. "If you take our platform and compare it to Cisco, the protocol features we provide and the hardware performance are all in the same range. The only difference is that the price is 40 percent to 60 percent lower."
Though Pica8 spun off of Quanta, Liao says that the company will also sell switches from other ODMs. But he declined to name them. But he does say Pica8 is selling gear to Japanese telecom giant NTT and Baidu, the company that dominates the Chinese search engine market.
Matthias Machowinski, of research firm Infonetics, says he is "very much aware" of this trend, though he adds that it is extremely hard to track. He says that the big web giants account for only a part of the overall switch market -- "the number of customers that might choose to go down this route are very limited. Today, you can count them on one hand, and maybe over the next two years, two hands might be enough" -- but he also acknowledges that as businesses move their applications onto services such as Amazon EC2 and Microsoft Azure -- rather than running stuff in their own data centers -- these web giants will account for an even larger part of the switch market.
Like Server, Like Switch
This shadow networking market is a repeat of what happened in the server world. Years ago, Google started building its own servers in tandem with the Asian ODMs, and other web giants followed. These companies are looking to save cost, but they're also looking to reduce their power consumption, customizing machines so they're far more efficient than their mass-market brethren.
In 2009, Google revealed some server designs it produced several years before. But, as with networking practices, the company says very little about its server gear. Amazon operates in much the same way. But Facebook had taken a different approach. Last year, after building its own data centers and working with various manufacturers to build its own servers, the social networking giant open sourced these designs to the rest of the world, hoping that others across the industry can help improve those designs, buy more hardware based on the designs, and ultimately drive down the price of the hardware.
>"It's kind like buying couches. If you buy one, you go to a retail store. If you buy 10,000 couches, you go straight to the factory"
This Open Compute Project already has several other big-name backers, including Texas-based cloud computing outfit Rackspace and Japan's NTT. And it doesn't stop at data centers and servers. Last month, Frank Frankovsky -- the ex-Dell man who oversees hardware design at Facebook -- told us that the company is in the midst of building its own storage hardware and that these designs will be open sourced in early May.
In these cases, Facebook and Amazon and Google and others bypassed "original equipment manufacturers," or OEMs, such as Dell and HP. The servers sold by the likes of HP and Dell are actually manufactured by those same ODMs in Taiwan and China.
James Liao, of Pica8 and formerly of Quanta, does not work with servers. But he says that it's common knowledge that -- like Google and Facebook -- Amazon purchases at least some of its servers from ODMs in Asia. "For servers, Facebook and Amazon are taking almost exactly the same approach," he says. "Amazon also has some very high power designers, but they don’t do the design themselves. They come up with a certain architecture and they tell the ODMs: 'This is my vision. These are the goals. And I want help designing the hardware.'"
Now, Liao says, this same sort of thing is happening with, well, everything. "All of the data center hardware is bought this way," Liao says. "You can refer to Facebook as an example, where one of the big projects inside the Open Compute effort is storage. Even the storage side is being commoditized. Servers, storage, and networking -- all of them are going to this way."
Howard Wu -- the president of greater China for Joyent, an Amazon-like cloud computing outfit based in San Francisco -- agrees. "If you're a small business and you're going to buy five servers, you're going to Dell or HP, because of the support services. But if you're a data center operator and you're going to buy 10,000 servers, you're going straight [to the ODMs]," he says. "It's kind like buying couches. If you buy one, you go to a retail store. If you buy 10,000 couches, you go straight to the factory."
That said, Joyent is not yet buying its gear from the ODMs.
"We are definitely in talks, but it hasn't actually happened yet," he says. "We have other contractual obligations right now." The market has not completely shifted to Asia. It's moving in stages. These web companies have many suppliers -- that's just good for businesses -- and in some cases, they're still buying hardware from the traditional players -- perhaps because they still have contracts in place. Facebook, for instance, is still buying some servers from Dell and HP. And Amazon is still buying custom servers from Rackable, a stateside manufacturer, and apparently other outfits based here in America.
The hardware supply chain is vast and varied. But it's consolidating. Now that they have the engineering talent, J.R. Rivers says, the ODMs are transforming into OEMs. "The market is maturing to the point where anyone can buy directly from ODMs," he says. "You don't have to be Google."
Update: In response to this story -- which refers to Pica8 as a spin-off of Quanta -- Quanta sent Wired a statement that reads: "Pica8 has licensed technology from Quanta and is not a "'spin off' of Quanta. Quanta has no ownership interest in Pica8 and has never owned Pica8's shares."
