Skip to main content
SpringerLink
Log in

A survey of memory deduplication approaches for intelligent urban computing

  • Special Issue Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Limited memory size is considered as a major bottleneck in data centers for intelligent urban computing. It is shown that there exist a large number of duplicated pages when various processes working with big data are hosted in data centers. Memory deduplication aims to automatically eliminate duplicate data in memory. It is an efficient technique to reduce memory requirement. In memory deduplication, pages with same content are detected and merged into a single copy. Recently, several system-level techniques have been proposed to address this issue, in which content-based page sharing (CBPS) is most widely used, since CBPS can be performed transparently in the hypervisor layer without any modification to guest operating systems of data center. In this paper, we survey several techniques for memory deduplication. We also classify these techniques based on their characteristics to highlight their similarities and differences. The aim of this paper is to provide insights to researchers into working of memory deduplication techniques and also to motivate them to propose better intelligent urban computing systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Xia, W., Jiang, H., Feng, D., Hua, Y.: SiLo: A similarity-locality based near-exact deduplication scheme with low ram overhead and high throughput. In: USENIX Annual Technical Conference (2011)

  2. Yan, C., Zhang, Y., Dai, F., Wang, X., Li, L., Dai, Q.: Parallel deblocking filter for HEVC on many-core processor. Electron. Lett. 50(5), 367–368 (2014)

    Article  Google Scholar 

  3. Arcangeli, A., Eidus, I., Wright, C.: Increasing memory density by using KSM. In: Proceedings of the Linux Symposium pp. 19-28 (2009)

  4. Yan, C., Zhang, Y., Dai, F., Li, L.: Highly parallel framework for HEVC motion estimation on many-core platform. In: Data Compression Conference (DCC), 2013, pp. 63–72. IEEE (2013)

  5. Yan, C., Zhang, Y., Xu, J., Dai, F., Zhang, J., Dai, Q., Wu, F.: Efficient parallel framework for HEVC motion estimation on many-core processors. IEEE Trans. Circuits Syst. Video Technol. 24(12), 2077–2089 (2014)

    Article  Google Scholar 

  6. Chang, C.R., Wu, J.J., Liu, P.: An empirical study on memory sharing of virtual machines for server consolidation. In: 2011 IEEE 9th International Symposium on Parallel and Distributed Processing with Applications (ISPA), pp. 244–249. IEEE (2011)

  7. Yan, C., Zhang, Y., Xu, J., Dai, F., Li, L., Dai, Q., Wu, F.: A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors. IEEE Signal Process. Lett. 21(5), 573–576 (2014)

    Article  Google Scholar 

  8. Gupta, D., Lee, S., Vrable, M., Savage, S., Snoeren, A.C., Varghese, G., Vahdat, A.: Difference engine: harnessing memory redundancy in virtual machines. Commun. ACM 53(10), 85–93 (2010)

    Article  Google Scholar 

  9. Yan, C., Zhang, Y., Dai, F., Zhang, J., Li, L., Dai, Q.: Efficient parallel HEVC intra-prediction on many-core processor. Electron. Lett. 50(11), 805–806 (2014)

    Article  Google Scholar 

  10. Sindelar, M., Sitaraman, R. K., Shenoy, P.: Sharing-aware algorithms for virtual machine colocation. In: Proceedings of the Twenty-Third Annual ACM Symposium on Parallelism in Algorithms and Architectures, pp. 367–378. ACM (2011)

  11. Veni, T., Bhanu, S.M.S.: Enhancing memory deduplication using temporal page sharing behaviors in virtual environments. In: Kumar Kundu, M., Mohapatra, D., Konar, A., Chakraborty, A. (eds.) Advanced Computing, Networking and Informatics-Volume 2, pp. 481–488. Springer International Publishing, Cham (2014)

    Chapter  Google Scholar 

  12. Zhang, D., Han, J., Li, C., Wang, J., Li, X.: Detection of co-salient objects by looking deep and wide. Int. J. Comput. Vis. 120(2), 215–232 (2016)

    Article  MathSciNet  Google Scholar 

  13. Zhang, D., Han, J., Jiang, L., Ye, S., Chang, X.: Revealing event saliency in unconstrained video collection. IEEE Trans. Image Process. 26(4), 1746–1758 (2017)

    Article  MathSciNet  Google Scholar 

  14. Cheng, G., Zhou, P., Han, J.: Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Trans. Geosci. Remote Sens. 54(12), 7405–7415 (2016)

    Article  Google Scholar 

  15. Yan, C., et al.: Supervised hash coding with deep neural network for environment perception of intelligent vehicles. Trans. Intell. Transp. Syst. 2017. IEEE (2017)

  16. Yan, C., et al: Effective Uyghur language text detection in complex background images for traffic prompt identification. Trans. Intell. Transp. Syst. 2017. IEEE (2017)

  17. Deng, Y., Hu, C., Wo, T., Li, B., Cui, L.: A memory deduplication approach based on group in virtualized environments. In: 2013 IEEE 7th International Symposium on Service Oriented System Engineering (SOSE), pp. 367-372. IEEE (2013)

  18. Miłós, G., Murray, D. G., Hand, S., Fetterman, M.A.: Satori: enlightened page sharing. In: Proceedings of the 2009 Conference on USENIX Annual Technical Conference, pp. 1–1 (2009)

  19. Brace, P.: Storage—moving towards tailored solutions, In: Non Volatiled Memory Workshop (NVMW) (2015)

  20. Goldberg, R.P.: Survey of virtual machine research. Computer 7(6), 34–45 (1974)

    Article  Google Scholar 

  21. Kloster, J.F., Kristensen, J., Mejlholm, A.: Determining the use of interdomain shareable pages using kernel introspection. Department of Computer Science, Aalborg University (2007)

  22. Sharma, P., Kulkarni, P.: Singleton: system-wide page deduplication in virtual environments. In: Proceedings of the 21st International Symposium on High-Performance Parallel and Distributed Computing, pp. 15–26. ACM (2012)

  23. Lin, X., Douglis, F., Li, J., Li, X., Ricci, R., Smaldone, S., Wallace, G.: Metadata considered harmful... to deduplication. In: HotStorage. USENIX Association, Santa Clara, CA (2015)

  24. Van Moolenbroek, D.C., Appuswamy, R., Tanenbaum, A.S.: Towards a flexible, lightweight virtualization alternative. In: Proceedings of International Conference on Systems and Storage, pp. 1–7. ACM (2014)

  25. Wood, T., Tarasuk-Levin, G., Shenoy, P., Desnoyers, P., Cecchet, E., Corner, M.D.: Memory buddies: exploiting page sharing for smart colocation in virtualized data centers. In: Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, pp. 31–40. ACM (2009)

  26. Xia, L., Dinda, P.A.: A case for tracking and exploiting inter-node and intra-node memory content sharing in virtualized large-scale parallel systems. In: Proceedings of the 6th International Workshop on Virtualization Technologies in Distributed Computing Date, pp. 11–18. ACM (2012)

  27. Nam, Y., Lee, D., Eom, Y.I.: SELF: Improving the memory-sharing opportunity using virtual-machine self-hints in virtualized systems. In: Proceedings of the 6th Asia-Pacific Workshop on Systems, p. 10. ACM (2015)

  28. Xiao, J., Xu, Z., Huang, H., Wang, H.: Security implications of memory deduplication in a virtualized environment. In: 2013 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 1–12. IEEE (2013)

  29. Pham, B., Veselý, J., Loh, G.H., Bhattacharjee, A.: Large pages and lightweight memory management in virtualized environments: Can you have it both ways?. In: Proceedings of the 48th International Symposium on Microarchitecture, pp. 1–12. ACM (2015)

  30. Kloster, J.F., Kristensen, J., Mejlholm, A., Behrmann, G.: On the feasibility of memory sharing: content-based page sharing in the xen virtual machine monitor. Aalborg University (2006)

  31. Miller, K., Franz, F., Groeninger, T., Rittinghaus, M., Hillenbrand, M., Bellosa, F.: KSM\(++\): using I/O-based hints to make memory-deduplication scanners more efficient. In: Proceedings of the ASPLOS Workshop on Runtime Environments, Systems, Layering and Virtualized Environments (RESoLVE’12) (2012)

  32. Miller, K., Franz, F., Rittinghaus, M., Hillenbrand, M., Bellosa, F.: XLH: more effective memory deduplication scanners through cross-layer hints. In: USENIX Annual Technical Conference, pp. 279–290 (2013)

  33. Mandal, S., Kuenning, G., Ok, D., Shastry, V., Shilane, P., Zhen, S., Zadok, E.: Using hints to improve inline block-layer deduplication. In: FAST, pp. 315–322 (2016)

  34. Rachamalla, S., Mishra, D., Kulkarni, P.: All page sharing is equal, but some sharing is more equal than others. http://www.cse.iitb.ac.in/internal/techreports/reports/TR-SE-2013-49.pdf

  35. Lee, S., Kim, I., Lee, D., Eom, Y.I.: The page cache deduplication mechanism in virtualized systems. IJCA 2013 6, 151–160 (2013)

    Google Scholar 

  36. Huang, Y., Yang, R., Cui, L., Wo, T., Hu, C., Li, B.: Vmcsnap: taking snapshots of virtual machine cluster with memory deduplication. In: 2014 IEEE 8th International Symposium on Service Oriented System Engineering (SOSE), pp. 314–319. IEEE (2014)

  37. Chen, L., Wei, Z., Cui, Z., Chen, M., Pan, H., Bao, Y.: CMD: classification-based memory deduplication through page access characteristics. ACM SIGPLAN Not. 49(7), 65–76 (2014)

    Article  Google Scholar 

  38. Korb, I., Kotthaus, H., Marwedel, P.: mmapcopy: efficient memory footprint reduction using application knowledge. In: Proceedings of the 31st Annual ACM Symposium on Applied Computing, pp. 1832–1837. ACM (2016)

  39. Kim, S.H., Jeong, J., Lee, J.: Selective memory deduplication for cost efficiency in mobile smart devices. IEEE Trans. Consum Electron. 60(2), 276–284 (2014)

    Article  Google Scholar 

  40. Bugnion, E., Devine, S., Govil, K., Rosenblum, M.: Disco: Running commodity operating systems on scalable multiprocessors. ACM Trans. Comput. Syst. (TOCS) 15(4), 412–447 (1997)

  41. Yang, X., Ye, C., Lin, Q.: Evaluation and enhancement to memory sharing and swapping in Xen 4.1. Xen Summit (2011)

  42. Barker, S.K., Wood, T., Shenoy, P.J., Sitaraman, R.K.: An empirical study of memory sharing in virtual machines. In: USENIX Annual Technical Conference, pp. 273-284 (2012)

  43. Kotthaus, H., Korb, I., Engel, M., Marwedel, P.: Dynamic page sharing optimization for the R language. ACM SIGPLAN Not. 50(2), 79–90 (2015)

  44. Chiang, J.H., Li, H.L., Chiueh, T.C.: Introspection-based memory de-duplication and migration. ACM SIGPLAN Not. 48(7), 51–62 (2013). ACM

    Article  Google Scholar 

  45. Tarasov, V., Jain, D., Kuenning, G., Mandal, S., Palanisami, K., Shilane, P., Zadok, E.: Dmdedup: device mapper target for data deduplication. In: Ottawa Linux Symposium, Ottawa (2014)

  46. Sha, S., Li, J., Li, N., Ju, W., Cui, L., Li, B.: Smartksm: a vmm-based memory deduplication scanner for virtual machines. http://act.buaa.edu.cn/lijx/pubs/sosp2013.smartksm.pdf

  47. Lim, K., Turner, Y., Santos, J.R., AuYoung, A., Chang, J., Ranganathan, P., Wenisch, T. F.: System-level implications of disaggregated memory. In: 2012 IEEE 18th International Symposium on High Performance Computer Architecture (HPCA), pp. 1–12. IEEE (2012)

  48. Lee, B., Kim, S.M., Park, E., Han, D.: MemScope: analyzing memory duplication on android systems. In: Proceedings of the 6th Asia-Pacific Workshop on Systems, p. 19. ACM (2015)

  49. Ceron, R., Folco, R., Leitao, B., Tsubamoto, H.: Power Systems Memory Deduplication. IBM Redbooks (2012)

Download references

Acknowledgements

This work is supported by National Nature Science Foundation of China (61671196, 61327902), Zhejiang Province Nature Science Foundation of China LR17F030006.

Author information

Authors and Affiliations

  1. Institute of Information and Control Hangzhou Dianzi University, Hangzhou, 310018, China

    Hailiang Huang, Chenggang Yan & Bingtao Liu

  2. Institute of Computing Technology, Chinese Academy of Science, Beijing, 100190, China

    Licheng Chen

Authors
  1. Hailiang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chenggang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bingtao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Licheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chenggang Yan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, H., Yan, C., Liu, B. et al. A survey of memory deduplication approaches for intelligent urban computing. Machine Vision and Applications 28, 705–714 (2017). https://doi.org/10.1007/s00138-017-0834-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-017-0834-6

Keywords

Search

Navigation