Skip to main content
SpringerLink
Log in

A Reliable Replica Mechanism for Stream Processing

  • Conference paper
  • First Online:
Collaborate Computing: Networking, Applications and Worksharing (CollaborateCom 2016)

  • 1710 Accesses

Abstract

In Internet of Things, data would be fast generated from massive sensors as real-time data stream, and the replica mechanism is essential to guarantee availability during stream processing. Traditional mechanisms always assume the redundant replicas were exactly correct, but in the practical conditions even slight errors of replica would lead to the calamity for recovery. In this paper, a reliable mechanism is proposed in which space-bounded signature of checkpoint is used for validation during the replica placement. The mechanism has been analyzed theoretically, and also demonstrated by extensive experiments in various conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ding, W., Han, Y., Wang, J., Zhao, Z.: Feature-based high-availability mechanism for quantile tasks in real-time data stream processing. Softw. Pract. Experience 44, 855–871 (2014)

    Article  Google Scholar 

  2. Bockermann, C.: A Survey of the Stream Processing Landscape. Lehrstuhl furk unstliche Intelligenz Technische Universität Dortmund (2014)

    Google Scholar 

  3. Barlow, M.: Real-Time Big Data Analytics: Emerging Architecture. O’Reilly Media Inc., Sebastopol (2013)

    Google Scholar 

  4. Ding, W., Han, Y., Wang, J., Zhao, Z.: Feature-based high availability mechanism for extreme aggregation tasks in real-time data stream processing. J. Internet Technol. 14, 327–340 (2013)

    Google Scholar 

  5. Hwang, J.H., Balazinska, M., Rasin, A., Cetintemel, U., Michael, S., Stan, Z.: High-availability algorithms for distributed stream processing. In: The 21st International Conference on Data Engineering, pp. 779–790 (2005)

    Google Scholar 

  6. Balazinska, M., Balakrishnan, H., Madden, S.R., Stonebraker, M.: Fault-tolerance in the borealis distributed stream processing system. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 13–24. ACM (2005)

    Google Scholar 

  7. Liu, Q., Lui, J.C., He, C., Pan, L., Fan, W., Shi, Y.: SAND: a fault-tolerant streaming architecture for network traffic analytics. In: The 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2014), Atlanta, Georgia, USA, pp. 80–87 (2014)

    Google Scholar 

  8. Akidau, T., Balikov, A., Bekiroglu, K., Chernyak, S., Haberman, J., Lax, R., McVeety, S., Mills, D., Nordstrom, P., Whittle, S.: MillWheel: fault-tolerant stream processing at internet scale. Proc. VLDB Endow. 6, 1033–1044 (2013)

    Article  Google Scholar 

  9. http://archive.cloudera.com/cdh/3/flume/

  10. Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., Bhagat, N., Mittal, S., Ryaboy, D.: Storm@twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 147–156. ACM, Snowbird (2014)

    Google Scholar 

  11. Gu, Y., Zhang, Z., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: An empirical study of high availability in stream processing systems. In: Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware, pp. 1–9. Springer, New York (2009)

    Google Scholar 

  12. Miller, G.L.: Riemann’s hypothesis and tests for primality. J. Comput. Syst. Sci. 13, 300–317 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  13. Rabin, M.O.: Probabilistic algorithm for testing primality. J. Number Theory 12, 128–138 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  14. Agrawal, M., Kayal, N., Saxena, N.: PRIMES is in P. Ann. Math. 160, 781–793 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  15. http://primes.utm.edu/lists/small/millions/

  16. http://en.wikipedia.org/wiki/Prime_number_theorem

  17. Ding, W., Zhao, Z., Han, Y.: A framework to improve the availability of stream computing. In: 23rd IEEE International Conference on Web Services (ICWS 2016), pp. 594–601. IEEE (2016)

    Google Scholar 

Download references

Acknowledgment

This work was supported by the R&D General Program of Beijing Education Commission (No. KM2015_10009007), the Key Young Scholars Foundation for the Excellent Talents of Beijing (No. 2014000020124G011) and Foundation for the Excellent Youth Scholars of North China University of Technology (XN072-006).

Author information

Authors and Affiliations

  1. Data Engineering Institute, North China University of Technology, Beijing, China

    Weilong Ding & Zhuofeng Zhao

  2. Beijing Key Laboratory on Integration and Analysis of Large-Scale Stream Data, Beijing, China

    Yanbo Han

Authors
  1. Weilong Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhuofeng Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanbo Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weilong Ding .

Editor information

Editors and Affiliations

  1. Beijing University of Posts and Telecommunications, Beijing, China

    Shangguang Wang

  2. Beijing University of Posts and Telecommunications, Beijing, China

    Ao Zhou

Rights and permissions

Reprints and permissions

Copyright information

© 2017 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Cite this paper

Ding, W., Zhao, Z., Han, Y. (2017). A Reliable Replica Mechanism for Stream Processing. In: Wang, S., Zhou, A. (eds) Collaborate Computing: Networking, Applications and Worksharing. CollaborateCom 2016. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 201. Springer, Cham. https://doi.org/10.1007/978-3-319-59288-6_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-59288-6_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-59287-9

  • Online ISBN: 978-3-319-59288-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation