{"id":1020,"date":"2013-09-26T00:54:00","date_gmt":"2013-09-25T16:54:00","guid":{"rendered":"http:\/\/www.brofive.org\/?p=1020"},"modified":"2018-04-29T01:39:29","modified_gmt":"2018-04-28T17:39:29","slug":"hadoop-at-yahoo-sets-new-gray-sort-recordzz","status":"publish","type":"post","link":"http:\/\/www.brofive.net\/?p=1020","title":{"rendered":"Hadoop at Yahoo! Sets New Gray Sort Record[zz]"},"content":{"rendered":"
\u53bb\u5e74\u66fe\u7ecf\u6574\u7406\u4e86\u6240\u6709Terasort\u7684\u8bb0\u5f55<\/a>\uff0c\u6700\u5feb\u7684\u662fYahoo 2009\u5e74\u505a\u7684TB\u6392\u5e8f\uff0c62\u79d2\u949f\uff081460\u8282\u70b9\uff0c\u4e3b\u98912.5Ghz\uff0c8G\uff0c1Gbe\uff0c4HD\uff09\u3002\u4eca\u5e747\u6708\u4efdYahoo\u5c31\u641e\u4e86\u4e00\u4e2a100TB\u7684\u6d4b\u8bd5\uff0c72\u5206\u949f\uff082100\u8282\u70b9\uff0c\u4e3b\u98912.3Ghz\uff0c\u53cc\u8def6\u6838\uff0c64G\uff0c10Gbe\uff0c12HD\uff09\u3002\u5982\u679c\u662f\u7ebf\u6027\u7684\uff0c\u52191TB\u7684\u6570\u636e\u6269\u5927\u5230100TB\uff0c\u9700\u8981103\u5206\u949f\uff0c\u5f53\u7136\uff0c\u6392\u5e8f\u7b97\u6cd5\u5e76\u975e\u7ebf\u6027\u7684\u3002\u4eca\u5e74Yahoo\u8fbe\u5230\u8fd9\u4e48\u9ad8\u7684\u6027\u80fd\uff0c\u867d\u7136\u548cHadoop\u4f18\u5316\u6709\u5173\u7cfb\uff0c\u4f46\u662f\u548c\u91c7\u7528\u9ad8\u6027\u80fd\u670d\u52a1\u5668\u7684\u5173\u7cfb\u66f4\u5927\u3002<\/h5>\n
Hadoop at Yahoo! Sets New Gray Sort Record \u2013 The Yellow Elephant is Getting Faster<\/h5>\n

By Thomas Graves \u2013 Wed, Jul 3, 2013 5:27 AM EDT<\/abbr><\/cite> <\/p>\n

hadoop-elephantWe are proud to announce we used Apache Hadoop<\/a> to set a new Gray sort record for the Jim Gray’s Sort<\/a> benchmark. We nearly doubled the rate of the previous Gray sort entry by sorting at a rate of 1.42 Terabytes per minute. The previous record was 0.725 Terabytes per minute. <\/p>\n

Jim Gray’s sort benchmark consists of a set of many related benchmarks, each with their own rules. All of the sort benchmarks measure the time to sort different numbers of 100 byte records. The first 10 bytes of each record is the key and the rest is the value. The Gray sort<\/strong> is to measure the sort rate achieved while sorting at least 100 terabytes of data. The Minute sort<\/strong> is the amount of data that can be sorted in less than a minute. There are two different benchmark categories. The Daytona category requires the sort code to be general purpose sort. The Indy category needs to only sort 100-byte records with 10-byte keys. We used Hadoop Terasort with slightly different configurations in both categories. <\/p>\n

There were some new rules this year. The biggest rule change is that your sort is required to sort both skewed and non-skewed data for the Daytona benchmark category. The skewed data has to be sorted in no more than twice the elapsed time of the non-skewed data. Other changes include: the input and output data must persist in the case of a single node failure and none of the data can be compressed (input, intermediate, or output) <\/p>\n

Results<\/strong> <\/p>\n

Our only official entry was to Gray sort, but we also unofficially (we didn\u2019t get the submission in by the deadline – learned that lesson!) broke the previous Minute Sort<\/strong> record. The full report can be found on the sort benchmark<\/a> page under the Gray sort results. <\/p>\n

\"21\"<\/a> <\/p>\n

Software, Hardware, and Operating System<\/strong> <\/p>\n

The version of Hadoop used was Hadoop 0.23.7. Hadoop 0.23.7 is an early branch of the Hadoop 2.X line that Yahoo! has used to stabilize YARN. It is available for download at hadoop.apache.org<\/a>. <\/p>\n

The hardware and operating system details are: <\/p>\n