skip to main content
10.1145/2588555.2593662acmconferencesArticle/Chapter ViewAbstractPublication PagesmodConference Proceedingsconference-collections
research-article

Patience is a virtue: revisiting merge and sort on modern processors

Published: 18 June 2014 Publication History

Abstract

The vast quantities of log-based data appearing in data centers has generated an interest in sorting almost-sorted datasets. We revisit the problem of sorting and merging data in main memory, and show that a long-forgotten technique called Patience Sort can, with some key modifications, be made competitive with today's best comparison-based sorting techniques for both random and almost sorted data. Patience sort consists of two phases: the creation of sorted runs, and the merging of these runs. Through a combination of algorithmic and architectural innovations, we dramatically improve Patience sort for both random and almost-ordered data. Of particular interest is a new technique called ping-pong merge for merging sorted runs in main memory. Together, these innovations produce an extremely fast sorting technique that we call P3 Sort (for Ping-Pong Patience+ Sort), which is competitive with or better than the popular implementations of the fastest comparison-based sort techniques of today. For example, our implementation of P3 sort is around 20% faster than GNU Quicksort on random data, and 20% to 4x faster than Timsort for almost sorted data. Finally, we investigate replacement selection sort in the context of single-pass sorting of logs with bounded disorder, and leverage P3 sort to improve replacement selection. Experiments show that our proposal, P3 replacement selection, significantly improves performance, with speedups of 3x to 20x over classical replacement selection.

References

[1]
David Aldous and Persi Diaconis. Longest increasing subsequences: from Patience sorting to the Baik-Deift-Johansson theorem. Bull. of the Amer. Math. Society, Vol. 36, No. 4, pages 413--432.
[2]
Sergei Bespamyatnikh and Michael Segal. Enumerating Longest Increasing Subsequences and Patience Sorting. Pacific Inst. for the Math. Sci. Preprints, PIMS-99--3., pp.7--8.
[3]
C. L. Mallows. "Problem 62--2, Patience Sorting". SIAM Review 4 (1962), 148--149.
[4]
Rajiv Wickremesinghe, Lars Arge, Jeffrey S. Chase, Jeffrey Scott Vitter: Efficient Sorting Using Registers and Caches. ACM Journal of Experimental Algorithmics 7: 9 (2002).
[5]
P. Gopalan et al. Estimating the Sortedness of a Data Stream. In SODA 2007.
[6]
A. LaMarca and R.E. Ladner. The influence of caches on the performance of sorting. Volume 7 (1997), pp. 370--379.
[7]
B. Chandramouli, J. Goldstein, and S. Duan. Temporal Analytics on Big Data for Web Advertising. In ICDE 2012.
[8]
TimSort. http://en.wikipedia.org/wiki/Timsort.
[9]
Sorting Algorithms. http://en.wikipedia.org/wiki/ Sort_algorithms.
[10]
R. Sedgewick. Implementing Quicksort programs. Comm. ACM 21 (10): 847--857.
[11]
GNU Quicksort Implementation. http://aka.ms/X5ho47.
[12]
Patience Sorting. http://en.wikipedia.org/wiki/ Patience_sorting.
[13]
Donald Knuth. The Art of Computer Programming, Sorting and Searching, Volume 3, 1998.
[14]
C.A.R. Hoare. Quicksort. Computer J. 5, 4, April 1962.
[15]
C.L. Mallows. Patience sorting. Bull. Inst. Math. Appl., 9:216--224, 1973.
[16]
J.M. Hammersley. A few seedlings of research. In Proc. Sixth Berkeley Symp. Math. Statist. and Probability, Volume 1, pages 345--394. University of California Press, 1972.
[17]
M. Kaufmann et al. Timeline Index: A Unified Data Structure for Processing Queries on Temporal Data in SAP HANA. In SIGMOD, 2013.
[18]
Splunk. http://www.splunk.com/.
[19]
K. E. Batcher. Sorting networks and their applications. In Spring Joint Computer Conference, pages 307--314, 1968.
[20]
J. Chhugani et al. Efficient Implementation of Sorting on MultiCore SIMD CPU Architecture. In VLDB, 2008.
[21]
P. Larson. External Sorting: Run Formation Revisited. IEEE Trans. Knowl. Data Eng. 15(4): 961--972 (2003).
[22]
C. Balkesen et al. Multi-Core, Main-Memory Joins: Sort vs. Hash Revisited. In VLDB, 2014.
[23]
M.-C. Albutiu et al. Massively parallel sort-merge joins in main memory multi-core database systems. In VLDB, 2012.
[24]
C. Kim et al. Sort vs. hash revisited: Fast join implementation on modern multi-core CPUs. In VLDB, 2009.

Cited By

View all
  • (2023)Cache-Efficient Top-k Aggregation over High Cardinality Large DatasetsProceedings of the VLDB Endowment10.14778/3636218.363622217:4(644-656)Online publication date: 1-Dec-2023
  • (2023)Optimal Influencer Marketing Campaign Under Budget Constraints Using Frank-WolfeIEEE Transactions on Network Science and Engineering10.1109/TNSE.2022.322595510:2(1015-1031)Online publication date: 1-Mar-2023
  • (2023)Backward-Sort for Time Series in Apache IoTDB2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00245(3196-3208)Online publication date: Apr-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGMOD '14: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data
June 2014
1645 pages
ISBN:9781450323765
DOI:10.1145/2588555
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2014

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. merging
  2. patience
  3. performance
  4. replacement selection
  5. sorting

Qualifiers

  • Research-article

Conference

SIGMOD/PODS'14
Sponsor:

Acceptance Rates

SIGMOD '14 Paper Acceptance Rate 107 of 421 submissions, 25%;
Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)27
  • Downloads (Last 6 weeks)1
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Cache-Efficient Top-k Aggregation over High Cardinality Large DatasetsProceedings of the VLDB Endowment10.14778/3636218.363622217:4(644-656)Online publication date: 1-Dec-2023
  • (2023)Optimal Influencer Marketing Campaign Under Budget Constraints Using Frank-WolfeIEEE Transactions on Network Science and Engineering10.1109/TNSE.2022.322595510:2(1015-1031)Online publication date: 1-Mar-2023
  • (2023)Backward-Sort for Time Series in Apache IoTDB2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00245(3196-3208)Online publication date: Apr-2023
  • (2022)APEXProceedings of the VLDB Endowment10.14778/3494124.349414115:3(597-610)Online publication date: 4-Feb-2022
  • (2020)Longest Increasing Subsequence under Persistent Comparison ErrorsTheory of Computing Systems10.1007/s00224-020-09966-9Online publication date: 20-Feb-2020
  • (2019)Strategies for stable merge sortingProceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms10.5555/3310435.3310513(1272-1290)Online publication date: 6-Jan-2019
  • (2018)Impatience Is a Virtue: Revisiting Disorder in High-Performance Log Analytics2018 IEEE 34th International Conference on Data Engineering (ICDE)10.1109/ICDE.2018.00067(677-688)Online publication date: Apr-2018
  • (2018)Longest Increasing Subsequence Under Persistent Comparison ErrorsApproximation and Online Algorithms10.1007/978-3-030-04693-4_16(259-276)Online publication date: 23-Aug-2018
  • (2017)A High Performance FPGA-Based Sorting Accelerator with a Data Compression MechanismIEICE Transactions on Information and Systems10.1587/transinf.2016EDP7383E100.D:5(1003-1015)Online publication date: 2017
  • (2017)A Memory Bandwidth-Efficient Hybrid Radix Sort on GPUsProceedings of the 2017 ACM International Conference on Management of Data10.1145/3035918.3064043(417-432)Online publication date: 9-May-2017
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media