http://lemire.me/blog/archives/2008/07/07/sorting-1-terabyte-in-209-seconds/ Computer Scientist and Open Scholar: Databases, Information Retrieval, Business Intelligence. Tue, 07 Feb 2012 23:39:16 +0000 hourly 1 http://wordpress.org/?v=3.3.1 http://lemire.me/blog/archives/2008/07/07/sorting-1-terabyte-in-209-seconds/comment-page-1/#comment-50008 Daniel Lemire Wed, 09 Jul 2008 22:29:47 +0000 http://www.daniel-lemire.com/blog/archives/2008/07/07/sorting-1-terabyte-in-209-seconds/#comment-50008 The 6000 seconds is definitively not "internal memory" since the whole machine has 2 GiB of RAM and it tries to sort 2 GiB of data. So there is quite a bit of IO overhead. Sure. The 6000 seconds is definitively not “internal memory” since the whole machine has 2 GiB of RAM and it tries to sort 2 GiB of data. So there is quite a bit of IO overhead. Sure.

]]> http://lemire.me/blog/archives/2008/07/07/sorting-1-terabyte-in-209-seconds/comment-page-1/#comment-50007 Kevembuangga Wed, 09 Jul 2008 19:29:45 +0000 http://www.daniel-lemire.com/blog/archives/2008/07/07/sorting-1-terabyte-in-209-seconds/#comment-50007 The Terabyte sort seem pretty silly, of course throwing a shitload of ressources at a problem is bound to give "impressive results" but where is the benefit for the average user? i.e your 6000 seconds sort. This looks like the Formula 1 racing which is supposed to further technological progress and which does once in a while, but at which cost? The <a href="http://www.hpl.hp.com/hosted/sortbenchmark/" rel="nofollow">Penny sort</a> on the same page seem more sensible. BTW, from experience with my linear sort the 6000 seconds you report for 2Gb fall within plausible range of elapsed time due to disk access latency when sorted records are shuffled around, <b>not</b> a compute bound limit, you might check it. The Terabyte sort seem pretty silly, of course throwing a shitload of ressources at a problem is bound to give “impressive results” but where is the benefit for the average user?
i.e your 6000 seconds sort.
This looks like the Formula 1 racing which is supposed to further technological progress and which does once in a while, but at which cost?
The Penny sort on the same page seem more sensible.
BTW, from experience with my linear sort the 6000 seconds you report for 2Gb fall within plausible range of elapsed time due to disk access latency when sorted records are shuffled around, not a compute bound limit, you might check it.

]]>