CNS Computer Scientists Claim World Data Sorting Record for Second Year
Not content to rest upon their laurels, a team of Center for Networked Systems (CNS) data center researchers broke two of their own world records set in 2010, and then succeeded in setting three more, when their system, Tritonsort-MR, sorted a terabyte (1 trillion bytes) of data in 106 minutes. The competition that they entered, the Sort Benchmark, is the large-scale data processing world’s Formula One World Championship and Daytona 500 rolled into one, and it attracts competitors from academic and industrial labs all over the world who vie to implement ever faster data center designs.
The CNS group consists of Dr. Amin Vahdat, Dr. George Porter, and Ph.D. students Alex Rasmussen and Michael Conley. Last year, they won in the “Indy” category for the “Gray” and “Minutesort” categories, racing to sort 1 TB of data as quickly as possible and as much data as possible in a single minute, respectively. The “Indy” category represents a parameter that exists only for the purpose of the competition, so that designing a system to compete here is comparable to constructing a racing vehicle that could only ever be driven on a track. But building on their successful foray in 2010, the team decided to take their game to a new level by adjusting their system to compete in the “Daytona,” or general purpose, category.
The key to the Tritonsort-MR design, says Porter, is seeking an efficient use of resources: “The whole aim of this project is to build balanced systems.” To do this, “we made some improvements on the data structures and algorithms- basically, to make it a lot more efficient in terms of sending records across the network.” The results from their new modifications in the “Daytona” general system not only were successful, but they also proved transferable to the original specialized system built to compete in the “Indy” category. Showing impressive improvements in performance, the team submitted for and won both categories in the “Gray” and “Minutesort” competitions. But beyond the achievement of speed, the efficiency of Tritonsort-MR’s design is remarkable: while the second place team used thirty-five hundred nodes to achieve their result, the Tritonsort-MR team used only fifty-two. If implemented in a real world data center, that would mean that Tritonsort-MR could allow a company to sort data more quickly while only making one-seventh of the investment in equipment, space, and energy costs for cooling and operation.
While winning in these four categories exceeded the team’s original goals from 2010, they found themselves intrigued by a new category on offer in 2011: the 100 Terabyte Joulesort competition in which teams vie to build a system that can sort the greatest number of data records while only consuming one joule of energy (for some idea of how much a joule is, it takes about a million Joules to watch TV for an hour). The introduction of this new category reflects the recognition of an increasingly dire challenge facing industry in trying to solve data intensive computing problems: energy usage. A primary reason that data centers are expensive to operate is because of the staggering scale of their energy consumption. Any design that could increase energy efficiency would have a positive and much needed impact on both the environment and on a company’s bottom line.
Though intrigued by this new opportunity, Rasmussen said that the team was skeptical at first that they could compete in Joulesort arena. “Typically when you look at systems that set records like this, they’re all built out of these incredibly energy efficient pieces.” However, the equipment typically used “you’d never see deployed in an actual data center setting” because of the considerable cost these sophisticated systems demand. The Tritonsort-MR team, on the other hand, was devoted more to the idea of making a system of direct applicability to enterprises with real-world needs and resources than to breaking a record for its own sake. This is reflected, said Rasmussen, in the type of equipment the Tritonsort-MR team employs for its system. “The stuff that we’re using is kind of commodity server stuff. We’ve got machines from HP that are a year and a half old” with multi-core processors and a Cisco Nexus 5596 switch. And as an additional challenge to the efficiency of the design, the team elected not to “do anything special as far as energy optimization is concerned.” But despite placing these limitations upon themselves, Tritonsort-MR won the Joulesort category handily. It was as if instead of building a racing vehicle the average person couldn’t afford and could only drive in special circumstances, the CNS team took a standard four door sedan and made it both remarkably fuel-efficient and the fastest car in the world.
Medals recognizing their accomplishments were awarded by the Sort Benchmark committee at the 2011 ACM SIGMOD conference.
The Tritonsort-MR team would like to acknowledge the support of CNS member company Cisco Systems, Inc. and the National Science Foundation.
