Skip to main content
SpringerLink
Log in

DL-Dedupe: Dual-Level Deduplication Scheme for Flash-Based SSDs

  • Conference paper
Web-Age Information Management (WAIM 2013)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 7901))

Included in the following conference series:

Abstract

Although flash memory-based Solid State Drive (SSD) was born as the replacement of Hard Disk Drive (HDD) due to its high performance and low power consumption, the limited write endurance of flash and the reliability problem hold back the footsteps of SSD popularization. We propose DL-Dedupe, a dual-level deduplication scheme for flash-based SSDs to reduce the amount of data written to flash memory and enhance the storage capacity at the same time by eliminating unnecessary replicated write operations, further reducing garbage collection to improve the lifespan of SSD. It combines chunk and page level deduplication to save memory utilization, reduce access latency and ensure deduplication effectiveness. It also introduces hybrid address mapping and adaptive cache replacement scheme to accelerate index lookup and reduce memory overhead for desirable access speed. The experimental results show that DL-Dedupe can significantly reduce redundant writes, thus enhance the reliability and lifespan of SSD.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Solid state 101 – an introduction to solid state storage (EB/OI) (April 17, 2009), http://www.snia.org/forums/sssi/SSSI.Wht.Paper.Final.pdf

  2. Yinjin, F., Hong, J., Nong, X., Lei, T., Fang, L.: AA-Dedupe: An Application-Aware Source Deduplication Approach for Cloud Backup Services in the Personal Computing Environment. In: CLUSTER 2011 (2011)

    Google Scholar 

  3. Richard, V., Carl, O., Matthew, E.: Big Data: What It Is and Why You Should Care. Framingham: IDC (White Paper), 1–14 (2011)

    Google Scholar 

  4. Quinlan, S., Dorward, S.: Venti: a new approach to archival storage. In: FAST 2002. USENIX Association (2002)

    Google Scholar 

  5. Kulkarni, P., Douglis, F.: Redundancy elimination within large collections of files. In: USENIX 2004. USENIX Association (2004)

    Google Scholar 

  6. Guo, F., Efstathopoulos, P.: Building a high-performance deduplication system. In: USENIX ATC 2011. USENIX Association (2011)

    Google Scholar 

  7. http://www.isuppli.com/Memory-and-Storage/MarketWatch/

  8. Lee, S., Park, D., Chung, T., Lee, D., Park, S., Song, H.: FAST: An FTL Scheme with Fully Associative Sector Translations. In: UKC (August 2005)

    Google Scholar 

  9. Lee, S., Shin, D.: LAST: locality-aware sector translation for NAND flash memory-based storage systems. SIGOPS 42(6) (2008)

    Google Scholar 

  10. Gupta, A., Kim, Y., Urgaonkar, B.: DFTL: A Flash Translation Layer Employing Demand-based Selective Caching of Page-level Address Mappings. In: ASPLOS 2009 (2009)

    Google Scholar 

  11. SNIA. IOTTA repository (January 2009), http://iotta.snia.org/

  12. Ricardo, K., Raju, R.: I/O Deduplication: Utilizing Content Similarity to Improve I/O Performance. In: FAST 2010. USENIX Association (2010)

    Google Scholar 

  13. Gupta, A., Pisolkar, R., Urgaonkar, B.: Leveraging Value Locality in Optimizing NAND Flash-Based SSDs. In: FAST 2011. USENIX Association (2011)

    Google Scholar 

  14. Debnath, B., Sengupta, S.: ChunkStash: Speeding up Inline Storage Deduplication using Flash Memory. In: USENIX ATC 2010 (2010)

    Google Scholar 

  15. Meister, D.: dedupv1: Improving Deduplication Throughput using Solid State Drives (SSD). In: MSST 2010 (2010)

    Google Scholar 

  16. Wu, G., He, X.: ΔFTL: Improving SSD Lifetime via Exploiting Content Locality. In: EuroSys 2012, Bern, Switzerland, April 10-13 (2012)

    Google Scholar 

  17. Chen, F., Luo, T., Zhang, X.: CAFTL: A content-aware flash translation layer enhancing the lifespan of flash memory based solid state drives. In: FAST 2011, San Jose, CA, February 15-17 (2011)

    Google Scholar 

  18. Kim, J., Lee, C., Lee, S.: Deduplication in SSDs: Model and Quantitative Analysis. In: MSST 2012 (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of High Performance Computing, National University of Defense Technology, Changsha, China

    Wanhui He, Nong Xiao, Fang Liu, Zhiguang Chen & Yinjin Fu

Authors
  1. Wanhui He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nong Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiguang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yinjin Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. College of Computer Science, Zhejiang University, Hangzhou, China

    Yunjun Gao

  2. Seoul National University, Seoul, Korea

    Kyuseok Shim

  3. Institute of Software, Chinese Academy of Sciences, South-Fourth-Street 4, Zhong-Guan-Cun, 100190, Beijing, P.R. China

    Zhiming Ding

  4. School of Computer Science and Technology, University of Science and Technology of China, 230027, Hefei, China

    Peiquan Jin

  5. School of Computer Science and Technology, Hangzhou Dianzi University, 310018, Hangzhou, China

    Zujie Ren

  6. Key Laboratory of Intelligence Computing and Novel Software Technology, Tianjin Key Laboratory of Computer Vision and System, Ministry of Education, Tianjin University of Technology, 300384, Tianjin, China

    Yingyuan Xiao

  7. CityU-USTC Advanced Research Institute, Suzhou, China

    An Liu

  8. School of Information Science and Technology, Southwest Jiaotong University, 610031, Chengdu, China

    Shaojie Qiao

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

He, W., Xiao, N., Liu, F., Chen, Z., Fu, Y. (2013). DL-Dedupe: Dual-Level Deduplication Scheme for Flash-Based SSDs. In: Gao, Y., et al. Web-Age Information Management. WAIM 2013. Lecture Notes in Computer Science, vol 7901. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39527-7_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-39527-7_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-39526-0

  • Online ISBN: 978-3-642-39527-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation