Skip to main content
SpringerLink
Log in

Sorting in a Memory Hierarchy with Flash Memory

  • Schwerpunktbeitrag
  • Published:
Datenbank-Spektrum Aims and scope Submit manuscript

Abstract

Flash memory continues to improve in price, capacity, reliability, durability, and performance. In addition to consumer and client devices, flash memory is also employed in many servers. Its optimal usage in various classes of server software, including web servers, file-and-print servers, and database servers, requires evaluation and analysis. The present paper analyzes the use of flash memory for database query processing including algorithms that combine flash memory and traditional disk drives.

External merge sort serves as a prototypical query execution algorithm. The excellent access latency of flash memory promises substantially better sort performance than sorting with traditional disks. Surprisingly, this is true only in fairly limited cases. Flash memory as intermediate external storage improves performance in particular in situations with very limited memory.

The most advantageous external sort algorithms combine flash memory and traditional disk, exploiting the fast access latency of flash memory as well as the fast transfer bandwidth and inexpensive capacity of traditional disks. Moreover, graceful degradation is required in multiple ways, both when spilling from memory to flash storage and when spilling from flash to disk. These algorithms are described in detail, including in particular their graceful transition from run generation to merging. Using such graceful degradation and incremental transitions, sorting in a three-level memory hierarchy of RAM, flash memory, and traditional disk can be generalized to database query processing with any number of levels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Recent high-end flash devices achieve a write bandwidth similar to their read bandwidth, presumably by keeping some space erased at all times ready for immediate writing. The bandwidth of flash devices and disks can be improved by parallelism or “arrays,” but those arrangements improve neither the cost-per-capacity of flash devices nor the access latency of disks.

  2. Sections 3.1 to 3.3 are heavily based on [12].

References

  1. Anon et al. (1985) A measure of transaction processing power. Datamation, 1 April 1985

  2. Andersen DG, Franklin J, Kaminsky M, Phanishayee A, Tan L, Vasudevan V (2009) FAWN: a fast array of wimpy nodes. In: Proceedings of the ACM SIGOPS 22nd symposium on operating systems principles, SOSP ’09. ACM, New York, pp 1–14

    Chapter  Google Scholar 

  3. Andreou P, Spanos O, Zeinalipour-Yazti D, Samaras G, Chrysanthis PK (2009) FSort: external sorting on flash-based sensor devices. In: DMSN

    Google Scholar 

  4. Bitton D, DeWitt DJ (1983) Duplicate record elimination in large data files. ACM TODS 8(2):255–265

    Article  MATH  Google Scholar 

  5. Bayer R, McCreight EM (1972) Organization and maintenance of large ordered indices. Acta Inform 1:173–189

    Article  Google Scholar 

  6. Dewitt DJ, Katz RH, Olken F, Shapiro LD, Stonebraker M, Wood DA (1984) Implementation techniques for main memory database systems. In: ACM SIGMOD, pp 1–8

    Google Scholar 

  7. Graefe G (1993) Query evaluation techniques for large databases. ACM Comput Surv 25(2). doi:10.1145/152610.152611

  8. Graefe G (2006) Implementing sorting in database systems. ACM Comput Surv 38(3). doi:10.1145/1132960.1132964

  9. Graefe G (2007) The five-minute rule twenty years later, and how flash memory changes the rules. In: DaMoN

    Google Scholar 

  10. Graefe G, Bunker R, Cooper S (1998) Hash joins and hash teams in Microsoft SQL Server. In: VLDB Conf, pp 86–97

    Google Scholar 

  11. Gray J, Graefe G (1997) The five-minute rule ten years later, and other computer storage rules of thumb. SIGMOD Rec 26(4):63–68

    Article  Google Scholar 

  12. Graefe G, Harizopoulos S, Kuno HA, Shah MA, Tsirogiannis D, Wiener JL (2010) Database operators for flash-enabled memory hierarchies. IEEE Data Eng Bull 33(4):21–27

    Google Scholar 

  13. Graefe G, Larson P-Å(2001) B-tree indexes and CPU caches. In: IEEE ICDE, pp 349–358

    Google Scholar 

  14. Graefe G, Linville A, Shapiro LD (1994) Sort versus hash revisited. IEEE Trans Knowl Data Eng 6(6):934–944

    Article  Google Scholar 

  15. Gray J, Putzolu GR (1987) The 5 minute rule for trading memory for disk accesses and the 10 byte rule for trading memory for CPU time. In: ACM SIGMOD, pp 395–398

    Google Scholar 

  16. Knuth DE (1973) The art of computer programming. Sorting and searching, vol III. Addison-Wesley, Reading

    Google Scholar 

  17. Lomet DB (2001) The evolution of effective B-tree page organization and techniques: a personal account. SIGMOD Rec 30(3):64–69

    Article  Google Scholar 

  18. Lee S-W, Moon B, Park C, Kim J-M, Kim S-W (2008) A case for flash memory ssd in enterprise database applications. In: SIGMOD, pp 1075–1086

    Chapter  Google Scholar 

  19. Nyberg C, Barclay T, Cvetanovic Z, Gray J, Lomet DB (1995) AlphaSort: a cache-sensitive parallel external sort. VLDB J 4(4):603–627

    Article  Google Scholar 

  20. Nakayama M, Kitsuregawa M, Takagi M (1988) Hash-partitioned join method using dynamic destaging strategy. In: VLDB, p 468

    Google Scholar 

  21. Ousterhout JK, Douglis F (1989) Beating the I/O bottleneck: a case for log-structured file systems. Oper Syst Rev 23(1):11–28

    Article  Google Scholar 

  22. Selinger PG, Astrahan MM, Chamberlin DD, Lorie RA, Price TG (1979) Access path selection in a relational database management system. In: ACM SIGMOD, pp 23–34

    Google Scholar 

  23. Zhang W, Larson P-Å(1997) Dynamic memory adjustment for external mergesort. In: VLDB Conf, pp 376–385

    Google Scholar 

Download references

Acknowledgements

Stavros Harizopoulos, Harumi Kuno, Mehul Shah, and Janet Wiener carefully read earlier drafts and provided many insightful comments. Editor and reviewers also contributed many pertinent and helpful suggestions.

Author information

Authors and Affiliations

  1. Hewlett-Packard Laboratories, Madison, WI, USA

    Goetz Graefe

Authors
  1. Goetz Graefe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Goetz Graefe.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Graefe, G. Sorting in a Memory Hierarchy with Flash Memory. Datenbank Spektrum 11, 83–90 (2011). https://doi.org/10.1007/s13222-011-0062-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13222-011-0062-6

Keywords

Search

Navigation