Chin-Hsien Wu
ABSTRACT
For in-memory computing frameworks such as Apache Spark [5, 6], objects (i.e., the intermediated data) can be accommodated in the main memory for speeding up the execution process. In this paper, we propose a cost-aware object management method for in-memory computing frameworks. When the main memory space of any worker node is not enough to accommodate the new computed or the retrieved object, we first pick appreciate objects which are already accommodated in the main memory as candidates for eviction and then evict objects with the minimal sum of the creation cost and the maximum sum of the occupied main memory space. According to the experimental results, we can achieve the goal under the 80/20 and 50/50 principles.
- 2014. Caching optimizing method of internal storage calculation. (March 12 2014). http://www.google.com/patents/CN103631730A?cl=en CN Patent App. CN 201,310,531,246.Google Scholar
- Mingxing Duan, Kenli Li, Zhuo Tang, Guoqing Xiao, and Keqin Li. 2015. Selection and replacement algorithms for memory performance improvement in Spark. Concurrency and Computation: Practice and Experience (2015), n/a-n/a. cpe.3584.Google Scholar
- Silvano Martello and Paolo Toth. 1990. Knapsack problems : algorithms and computer implementations. Toronto (Ont.), Chichester, New York. http://opac.inria.fr/record=b1088134 System requirements for computer disk: IBM PC; FORTRAN. Google ScholarDigital Library
- Tom White. 2009. Hadoop: The Definitive Guide (1st ed.). O'Reilly Media, Inc. Google ScholarDigital Library
- Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient Distributed Datasets: A Fault-tolerant Abstraction for In-memory Cluster Computing. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (NSDI'12). USENIX Association, Berkeley, CA, USA, 2--2. http://dl.acm.org/citation.cfm?id=2228298.2228301 Google ScholarDigital Library
- Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster Computing with Working Sets. In Proceedings of the 2Nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud'10). USENIX Association, Berkeley, CA, USA, 10--10. http://dl.acm.org/citation.cfm?id=1863103.1863113 Google ScholarDigital Library
Index Terms
- A cost-aware object management method for in-memory computing frameworks
Recommendations
Enabling Hybrid PCM Memory System with Inherent Memory Management
RACS '16: Proceedings of the International Conference on Research in Adaptive and Convergent SystemsReplacing the traditional volatile main memory, e.g., DRAM, with a non-volatile phase change memory (PCM) has become a possible solution to reduce the energy consumption of computing systems. To further reduce the bit cost of PCM, the development trend ...
Write-aware memory management for hybrid SLC-MLC PCM memory systems
In recent years, phase-change memory (PCM) has generated a great deal of interest because of its byte addressability and non-volatility properties. It is regarded as a good alternative storage medium that can reduce the performance gap between the main ...
File-Based Memory Management for Non-volatile Main Memory
COMPSAC '13: Proceedings of the 2013 IEEE 37th Annual Computer Software and Applications ConferenceActive research and development efforts on byte addressable non-volatile (NV) memory technologies, such as STT-RAM, PCM, and ReRAM, have been conducted in recent years. Because they are byte addressable, they can be used as main memory by directly ...
Comments