Skip to main content
SpringerLink
Log in

Achieving self-aware parallelism in stream programs

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The age of big data open the door to a new approach in data exploration and utilization. With the increasing complexities and dynamics of modern IT systems and services, it has become a challenge to effectively exploit parallelism on multicore platforms in computing systems that are heterogeneous, dynamic and decentralised. Self-aware software is a response to these demands in dealing with distributed applications in changing environments. It is a closed-loop system with a series of optimization strategies to adjust itself dynamicly during data processing. We focus on incorporating adaptation mechanisms into the stream programs for exposing distributed parallelism. In the traditional stream programming models, changing data and status normally require human supervision to adjust the stream graph for performance. As one-time optimization strategy, the reconfiguration and maintenance lead to costly and time-consuming procedures during the operating phase. To address these problems, we propose a self-aware stream programming model called StreamAware. A key property of this model is that exposing self-aware parallelism in the message driven execution paradigm, which provides dynamic and reconfigurable stream graph in adapting to the data flow changes. The model defines the self-awareness loop based on finite state machine for stream applications to adjust their own stream graph with continuous optimization strategy. This paper presents three different self-aware systems built using StreamAware. The empirical evaluation demonstrate how these systems can exploit self-aware parallelism using the Parsec benchmark problems, optimize performance per Watt, and respond to significant changes in stream processing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Wattsup.net.meter. (1997). http://www.wattsupmeters.com/

  2. Agarwala, S., Poellabauer, C., Kong, J., Schwan, K., Wolf, M.: Resource-aware stream management with the customizable dproc distributed monitoring mechanisms. In: Proceedings. 12th IEEE International Symposium on High Performance Distributed Computing, 2003, pp. 250–259. IEEE (2003)

  3. Ahmad, Y., Çetintemel, U.: Network-aware query processing for stream-based applications. In: Proceedings of the 13th International Conference on Very Large Data Bases-volume 30, pp. 456–467. VLDB Endowment (2004)

  4. Ashcroft, E.A., Wadge, W.W.: Lucid, a nonprocedural language with iteration. Commun. ACM 20(7), 519–526 (1977)

    Article  MATH  Google Scholar 

  5. Babaoglu, O.: Self-star Properties in Complex Information Systems: Conceptual and Practical Foundations, vol. 3460. Springer, Berlin (2005)

    Google Scholar 

  6. Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Proceedings of the 21st ACM SIGMOD-SIGACT-SIGART Symposium on Principles of database systems, pp. 1–16. ACM (2002)

  7. Barker, J.R., Tompkins, K.P.: Identification in the self-managing organization characteristics of target and tenure. Hum. Commun. Res. 21(2), 223–240 (1994)

    Article  Google Scholar 

  8. Bienia, C., Kumar, S., Singh, J.P., Li, K.: The parsec benchmark suite: Characterization and architectural implications. In: Proceedings of the 17th international conference on Parallel architectures and compilation techniques, pp. 72–81. ACM (2008)

  9. Boussinot, F., De Simone, R.: The esterel language. Proc. IEEE 79(9), 1293–1304 (1991)

    Article  Google Scholar 

  10. Buck, J.T.: Static scheduling and code generation from dynamic dataflow graphs with integer-valued control streams. In: 1994 Conference Record of the Twenty-Eighth Asilomar Conference on Signals, Systems and Computers, 1994. vol. 1, pp. 508–513. IEEE (1994)

  11. Buck, J.T., Lee, E.A.: Scheduling dynamic dataflow graphs with bounded memory using the token flow model. In: 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1993. ICASSP-93, vol. 1, pp. 429–432. IEEE (1993)

  12. Bunescu, R.C.: Learning with probabilistic features for improved pipeline models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 670–679. Association for Computational Linguistics (2008)

  13. Cao, F., Ester, M., Qian, W., Zhou, A.: Density-based clustering over an evolving data stream with noise. In: SDM. SIAM (2006)

  14. Cohen, A., Duranton, M., Eisenbeis, C., Pagetti, C., Plateau, F., Pouzet, M.: N-synchronous kahn networks: a relaxed model of synchrony for real-time systems. In: ACM SIGPLAN Notices, vol. 41, pp. 180–193. ACM (2006)

  15. Ding, L., Rundensteiner, E.A., Heineman, G.T.: Mjoin: a metadata-aware stream join operator. In: Proceedings of the 2nd International Workshop on Distributed Event-Based Systems, pp. 1–8. ACM (2003)

  16. Erez, M., Ahn, J.H., Gummaraju, J., Rosenblum, M., Dally, W.J.: Executing irregular scientific applications on stream architectures. In: Proceedings of the 21st Annual International Xonference on Supercomputing, pp. 93–104. ACM (2007)

  17. Golab, L., Özsu, M.T.: Issues in data stream management. ACM Sigmod Record 32(2), 5–14 (2003)

    Article  Google Scholar 

  18. Gordon, M.I., Thies, W., Amarasinghe, S.: Exploiting coarse-grained task, data, and pipeline parallelism in stream programs. SIGARCH Comput. Archit. News 34(5), 151–162 (2006)

    Article  Google Scholar 

  19. Grahne, G., Zhu, J.: Efficiently using prefix-trees in mining frequent itemsets. FIMI vol. 3, 123–132 (2003)

  20. Guha, S., Gunopulos, D., Koudas, N.: Correlating synchronous and asynchronous data streams. In: Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 529–534. ACM (2003)

  21. Gummaraju, J., Rosenblum, M.: Stream programming on general-purpose processors. In: Proceedings 38th Annual IEEE/ACM International Symposium on Microarchitecture, 2005. MICRO-38, p. 12. IEEE (2005)

  22. Gurout, T., Monteil, T., Costa, G.D., Calheiros, R.N., Buyya, R., Alexandru, M.: Energy-aware simulation with DVFS. Simul. Model. Pract. Theory 39, 76–91 (2013)

    Article  Google Scholar 

  23. Hernndez: Yet another survey on simd instructions (2013)

  24. Hoffmann, H., Maggio, M., Santambrogio, M.D., Leva, A., Agarwal, A.: Seec: A framework for self-aware computing (2010)

  25. Jacobs, S., Eleftheriadis, A.: Streaming video using dynamic rate shaping and tcp congestion control. J. Vis. Commun. Image Represent. 9(3), 211–222 (1998)

    Article  Google Scholar 

  26. Javadtalab, A., Omidyeganeh, M., Shirmohammandi, S., Hosseini, M.: On the suitability of current x264 rate controller algorithms for high definition video conferencing. In: 2011 International Symposium on Artificial Intelligence and Signal Processing (AISP), pp. 107–112 (2011)

  27. Kapasi, U.J., Dally, W.J., Rixner, S., Owens, J.D., Khailany, B.: The imagine stream processor. In: Proceedings. 2002 IEEE International Conference on Computer Design: VLSI in Computers and Processors, 2002, pp. 282–288. IEEE (2002)

  28. Kapasi, U.J., Rixner, S., Dally, W.J., Khailany, B., Ahn, J.H., Mattson, P., Owens, J.D.: Programmable stream processors. Computer 36(8), 54–62 (2003)

    Article  Google Scholar 

  29. Kephart, J.O., Chess, D.M.: The vision of autonomic computing. Computer 36(1), 41–50 (2003)

    Article  MathSciNet  Google Scholar 

  30. de Kock, E.A., Smits, W., van der Wolf, P., Brunel, J.Y., Kruijtzer, W., Lieverse, P., Vissers, K.A., Essink, G.: Yapi: application modeling for signal processing systems. In: Proceedings of the 37th Annual Design Automation Conference, pp. 402–405. ACM (2000)

  31. Kumar, V., Cooper, B.F., Cai, Z., Eisenhauer, G., Schwan, K.: Resource-aware distributed stream management using dynamic overlays. In: Proceedings of 25th IEEE International Conference on Distributed Computing Systems, 2005. ICDCS 2005, pp. 783–792. IEEE (2005)

  32. Laddaga, R.: Guest editor’s introduction: creating robust software through self-adaptation. IEEE Intell. Syst. 14(3), 0026–29 (1999)

  33. Lee, E., Messerschmitt, D.G.: Static scheduling of synchronous data flow programs for digital signal processing. IEEE Trans. Comput. 100(1), 24–35 (1987)

    Article  Google Scholar 

  34. Lewis, P.R., Chandra, A., Parsons, S., Robinson, E., Glette, K., Bahsoon, R., Torresen, J., Yao, X.: A survey of self-awareness and its application in computing systems. In: 2011 Fifth IEEE Conference on Self-Adaptive and Self-Organizing Systems Workshops (SASOW), pp. 102–107. IEEE (2011)

  35. Matko, D., Geiger, G., Gregoritza, W.: Pipeline simulation techniques. Math. Comput. Simul. 52(3), 211–230 (2000)

    Article  Google Scholar 

  36. Matko, D., Geiger, G., Gregoritza, W.: Verification of various pipeline models. Math. Comput. Simul. 53(4), 303–308 (2000)

    Article  Google Scholar 

  37. Mitchell, M.: Self-awareness and control in decentralized systems. In: AAAI Spring Symposium: Metacognition in Computation, pp. 80–85 (2005)

  38. Mondada, F., Pettinaro, G.C., Guignard, A., Kwee, I.W., Floreano, D., Deneubourg, J.L., Nolfi, S., Gambardella, L.M., Dorigo, M.: Swarm-bot: a new distributed robotic concept. Auton. Robots 17(2–3), 193–221 (2004)

    Article  Google Scholar 

  39. Murthy, P.K., Lee, E.A.: Multidimensional synchronous dataflow. IEEE Trans. Signal Process. 50(8), 2064–2079 (2002)

    Article  Google Scholar 

  40. Oreizy, P., Gorlick, M.M., Taylor, R.N., Heimhigner, D., Johnson, G., Medvidovic, N., Quilici, A., Rosenblum, D.S., Wolf, A.L.: An architecture-based approach to self-adaptive software. IEEE Intell. Syst. Appl. 14(3), 54–62 (1999)

    Article  Google Scholar 

  41. Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.: Network-aware operator placement for stream-processing systems. In: ICDE’06. Proceedings of the 22nd International Conference on Data Engineering, 2006, pp. 49–49. IEEE (2006)

  42. Rignot, E.: Changes in west antarctic ice stream dynamics observed with alos palsar data. Geophys. Res. Lett. 35(12), L12505 (2008)

    Google Scholar 

  43. Ruggiero, M., Guerri, A., Bertozzi, D., Poletti, F., Milano, M.: Communication-aware allocation and scheduling framework for stream-oriented multi-processor systems-on-chip. In: DATE’06. Proceedings of Design, Automation and Test in Europe, 2006, vol. 1, p. 6. IEEE (2006)

  44. Stroustrup, B.: C++11-the new iso c++ standard (2013). http://www.stroustrup.com/C++11FAQ.html

  45. Salehie, M., Tahvildari, L.: Self-adaptive software: landscape and research challenges. ACM Trans. Auton. Adapt. Syst. 4(2), 14 (2009)

    Article  Google Scholar 

  46. Schneider, D., Reich, W., Wiebel, A., Scheuermann, G.: Topology aware stream surfaces. In: Computer Graphics Forum, vol. 29, pp. 1153–1161. Wiley (2010)

  47. Sermulins, J., Thies, W., Rabbah, R., Amarasinghe, S.: Cache aware optimization of stream programs. In: ACM SIGPLAN Notices, vol. 40, pp. 115–126. ACM (2005)

  48. Sifakis, E., Neverov, I., Fedkiw, R.: Automatic determination of facial muscle activations from sparse motion capture marker data. In: ACM Transactions on Graphics (TOG), vol. 24, pp. 417–425. ACM (2005)

  49. Spring, J.H., Privat, J., Guerraoui, R., Vitek, J.: Streamflex: high-throughput stream programming in java. ACM SIGPLAN Notices 42(10), 211–228 (2007)

    Article  Google Scholar 

  50. Su, Y., Shi, F., Talpur, S., Wei, J., Tan, H.: Exploiting controlled-grained parallelism in message-driven stream programs. J. Supercomput. 70, 488–509 (2014)

    Article  Google Scholar 

  51. Tawney, G.: Feeling and self-awareness. Psychol. Rev. 9(6), 570 (1902)

    Article  Google Scholar 

  52. Thies, W., Karczmarek, M., Amarasinghe, S.: Streamit: a language for streaming applications. Compiler Construction, pp. 179–196. Springer, Berlin (2002)

    Chapter  Google Scholar 

  53. Vassev, E., Hinchey, M.: Knowledge representation and awareness in autonomic service-component ensembles-state of the art. In: 2011 14th IEEE International Symposium on Object/Component/Service-Oriented Real-Time Distributed Computing Workshops (ISORCW), pp. 110–119. IEEE (2011)

  54. Wei, T., Chen, X., Mishra, P.: Designing a multi-core hard real-time test bed for energy measurement experiments. In: Proceedings of the 2009 ACM symposium on Applied Computing, pp. 1998–1999. ACM (2009)

Download references

Author information

Authors and Affiliations

  1. Beijing Institute of Technology, Beijing, China

    Yan Su, Feng Shi, Yizhuo Wang, Sensen Hu & Jin Wei

  2. Mehran University of Engineering and Technology, Jamshor, Sindh, Pakistan

    Shahnawaz Talpur

Authors
  1. Yan Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shahnawaz Talpur
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yizhuo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sensen Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jin Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin Wei.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, Y., Shi, F., Talpur, S. et al. Achieving self-aware parallelism in stream programs. Cluster Comput 18, 949–962 (2015). https://doi.org/10.1007/s10586-014-0412-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-014-0412-x

Keywords

Search

Navigation