Skip to main content
SpringerLink
Log in

Building a Data Pipeline for the Management and Processing of Urban Data Streams

  • Chapter
  • First Online:
Book cover Handbook of Smart Cities

  • 1533 Accesses

Abstract

Urban data streams (UDS) originate from various sensors and Internet of Things (IoT) devices deployed in smart cities as well as social media sources such as Twitter and Facebook. The large volumes of urban data need to be harnessed to help smart city stakeholders and applications make informed decisions on the fly. Furthermore, effective management and governance of smart city components relies on the ability to integrate and federate their data, process urban data streams locally, and use big data analytics. Data integration and interoperability is a challenging problem that smart cities are facing today. Successful data integration is crucial for improved services and governance. This chapter describes a framework that aims to serve in building a data pipeline for the acquisition and processing of urban data streams, urban data analytics, and creation of value-added services. The framework relies on latest technologies for data processing including IoT, edge computing, data integration techniques, cloud computing, and data analytics. The proposed platform will facilitate real-time event detection, notification of alerts, mining the opinions of citizens regarding the governance of their city, and building monitoring dashboards. A prototype of the platform is being implemented using the Kafka messaging platform.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Anastasi G et al. (2013) Urban and social sensing for sustainable mobility in smart cities. in Proc. IFIP/IEEE Int. Conf. Sustainable Internet ICT Sustainability, Palermo, Italy, pp. 1–4.

    Google Scholar 

  2. Ciuccarelli P, Lupi G, Simeone L (2014) Visualizing the Data City: Social Media as a Source of Knowledge for Urban Planning and Management. Springer, Heidelberg.

    Book  Google Scholar 

  3. Cugola G. and Margara A (2012) Processing flows of information: from data stream to complex event processing. ACM Comput. Surv. 44 (3) 15:1–15:62.

    Article  Google Scholar 

  4. DataTorrent (2016) Real-Time Event Stream Processing – What are your choices?” https://www.datatorrent.com/blog/real-time-event-stream-processing-what-are-your-choices/. Latest access on Dec. 05. 2017.

  5. Hashem I A T, Chang V, Anuar N B, Adewole K, Yaqoob I, Gani A, Ahmed E, and Chiroma H (2016) The role of big data in smart city. International Journal of Information Management, vol. 36, no. 5, pp. 748–758.

    Article  Google Scholar 

  6. Jugel U, Jerzak Z, Hackenbroich G, and Markl V (2014) M4 - A Visualization-Oriented Time Series Data Aggregation. PVLDB, Volume 7 Issue 10, pp. 797–808.

    Google Scholar 

  7. Margara A, Urbani J, van Harmelen F, and Bal H (2014) Streaming the Web: Reasoning over dynamic data. Web Semantics: Science, Services and Agents on the World Wide Web, vol. 25, pp. 24–44.

    Article  Google Scholar 

  8. Ramparany F and Cao Q H (2016) A semantic approach to IoT data aggregation and interpretation applied to home automation. Internationl Conference on Internet of Things and Applications (IOTA), pp. 23–28.

    Google Scholar 

  9. Rosi A et al. (2011) Social sensors and pervasive services: Approaches and perspectives. Proc. IEEE Int. Conf. PERCOM Workshops, Seattle, WA, USA, pp. 525–530.

    Google Scholar 

  10. Tim L M V, Birte U, Vignesh S, Maria E. N (2014) Analyzing Tweets to Aid Situational Awareness. Advances in Information Retrieval, Vol. 8416, Lecture Notes in Computer Science, pp. 700–705.

    Article  Google Scholar 

  11. Vaccari A, Liu L, Biderman A, Ratti C, Pereira F, Oliveirinha J, Gerber A (2009) A holistic framework for the study of urban traces and the profiling of urban processes and dynamics. Proc. 12th International IEEE Conference on Intelligent Transportation Systems, IEEE Press, New York, pp. 273–278.

    Google Scholar 

  12. Hosebird Client (hbc). Retrieved from: https://github.com/twitter/hbc

  13. Barnaghi P, Tönjes R, Höller J, Hauswirth M, Sheth A T, Anantharam P. (2015) Citypulse: Real-time iot stream processing and large-scale data analytics for smart city applications. ict-citypulse.eu Deliverable D.3.2: Data Federation and Aggregation in Large-Scale Urban Data Streams.

    Google Scholar 

  14. Computing (2014) London Westminster City Council introduces smart parking system. https://www.computing.co.uk/ctg/news/2323408/london-westminster-city-council-introduces-smart-parking-system. Latest access on Dec. 05. 2017.

  15. Diginomica.com (2017) Bright Lights. Smart City. San Diego’s pioneering IoT platform. https://diginomica.com/2017/10/31/bright-lights-smart-city-san-diegos-pioneering-iot-platform/. Latest access on Dec. 05. 2017.

  16. Fraunhofer FOKUS Institute (2012) FixMyCity. https://www.fokus.fraunhofer.de/04c06110dd0adebc. Latest Access on Dec. 05. 2017.

  17. Gyrard A and Serrano M (2016) Connected Smart Cities - Interoperability with SEG 3.0 for the Internet of Things. Advanced Information Networking and Applications(AINA 2016) Workshops.

    Google Scholar 

  18. Haas L M, Lin E T, and Roth M T (2002) Data integration through database federation. IBM Systems Journal, 41(4), pp. 578–596.

    Article  Google Scholar 

  19. IDC.com (2017) IDC FutureScape: Worldwide Internet of Things 2017 Predictions. https://www.idc.com/research/viewtoc.jsp?containerId=US40755816. Latest Access on Dec. 05. 2017.

  20. Loshin D (2009) Data Consolidation and Integration. Master Data Management, Elsevier, pp. 177–199.

    Google Scholar 

  21. Trilles S, Calia A, Belmonte Ó, Torres-Sospedra J, Montoliu R, and Huerta J (2016) Deployment of an open sensorized platform in a smart city context. Future Generation Computer Systems, vol. 76, pp. 221–233.

    Article  Google Scholar 

  22. Wired.com (2015) The 20 Most Bike-Friendly Cities on the Planet. https://www.wired.com/2015/06/copenhagenize-worlds-most-bike-friendly-cities/ Latest access on Dec. 05. 2017.

  23. Aloi G et al. (2016) A Mobile Multi-Technology Gateway to Enable IoT Interoperability. Proc. IEEE First International Conference on Internet-of-Things Design and Implementation (IoTDI), pp. 259–264.

    Google Scholar 

  24. Aloi G, Caliciuri G, Fortino G, Gravina R, Pace P, Russo W, Savaglio C (2017) Enabling IoT interoperability through opportunistic smartphone-based mobile gateways. J.Netw.Comput.Appl. 81, pp. 73–83.

    Article  Google Scholar 

  25. Blackstock M and Lea R (2014) IoT interoperability: A hub-based approach. Proc. IEEE International Conference on the Internet of Things (IOT), pp. 79–84.

    Google Scholar 

  26. Asensio A, Marco A, Blasco R, Casas R (2014) Protocol and Architecture to Bring Things into Internet of Things. International Journal of Distributed Sensor Networks, Article ID 158252.

    Google Scholar 

  27. Blackstock M, Kaviani N, Lea R, and Friday A (2010) MAGIC Broker 2: An open and extensible platform for the Internet of Things. Proc. the 2010 Internet of Things (IOT), pp. 1–8.

    Google Scholar 

  28. Perera C, Jayaraman P P, Zaslavsky A, Christen P and Georgakopoulos D (2014) MOSDEN: An Internet of Things Middleware for Resource Constrained Mobile Devices. 47th Hawaii International Conference on System Sciences, Waikoloa, HI, pp. 1053–1062.

    Google Scholar 

  29. Feng G, Intizar Ali M, Curry E, and Mileo A (2017) Semantic Discovery and Integration of Urban Data Streams, Future Generation Computer Systems, Volume 76 Issue C, pp. 561–58.

    Google Scholar 

  30. Kettouch M, Luca C, Khorief O, Rui Wu and Dascalu S (2017) Semantic data management in Smart Cities. International Conference on Optimization of Electrical and Electronic Equipment (OPTIM 2017) & Intl Aegean Conference on Electrical Machines and Power Electronics (ACEMP 2017), pp. 1126–1131.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Information Technology, United Arab Emirates University, Al-Ain, United Arab Emirates

    Elarbi Badidi, Nouf El Neyadi, Meera Al Saeedi & Fatima Al Kaabi

  2. School of Computer Science, McGill University, Montreal, QC, Canada

    Muthucumaru Maheswaran

Authors
  1. Elarbi Badidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nouf El Neyadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meera Al Saeedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fatima Al Kaabi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Muthucumaru Maheswaran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elarbi Badidi .

Editor information

Editors and Affiliations

  1. School of Computer Science, McGill University, Montreal, QC, Canada

    Muthucumaru Maheswaran

  2. CIT, UAE University, Al Ain, United Arab Emirates

    Elarbi Badidi

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Badidi, E., El Neyadi, N., Al Saeedi, M., Al Kaabi, F., Maheswaran, M. (2018). Building a Data Pipeline for the Management and Processing of Urban Data Streams. In: Maheswaran, M., Badidi, E. (eds) Handbook of Smart Cities. Springer, Cham. https://doi.org/10.1007/978-3-319-97271-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-97271-8_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-97270-1

  • Online ISBN: 978-3-319-97271-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation