Abstract
In order to store and analyze the increasing data in recent years, big data techniques are applied to many fields such as healthcare, manufacturing, telecommunications, retail, energy, transportation, automotive, security, environment, etc. This work implements a city traffic state assessment system in cloud using a novel big data architecture. The proposed system provides the real-time busses location and real-time traffic state, especially the real-time traffic state nearby, through open data, cloud computing, bid data technology, clustering methods, and irregular moving average. With the high-scalability cloud technologies, Hadoop and Spark, the proposed system architecture is first implemented successfully and efficiently. Next, we utilize irregular moving average and clustering methods to find the area of traffic jam. Finally, three important experiments are performed. The first experiment indicates that the computing ability of Spark is better than that of Hadoop. The second experiment applies Spark to process bus location data under different number of executors. In the last experiment, we apply irregular moving average and clustering methods to efficiently find the area of traffic jam in Taiwan Boulevard which is the main road in Taichung city. Based on these experimental results, the provided system services are present via an advanced web technology.
Similar content being viewed by others
References
Wang, L., Lu, K., Liu, P., Ranjan, R., Chen, L.: IK-SVD: dictionary learning for spatial big data via incremental atom update. Comput. Sci. Eng. 16, 41–52 (2015a)
Wang, L., Geng, H., Liu, P., Lu, K., Kolodziej, J., Ranjan, R., Zomaya, A.Y.: Particle swarm optimization based dictionary learning for remote sensing big data. Knowl. Based Syst. 79, 43–50 (2015b)
Ma, Y., Wang, L., Liu, P., Ranjan, R.: Towards building a data-intensive index for big data computing—a case study of remote sensing data processing. Inf. Sci. 319, 171–188 (2015)
Ibm smarter business. http://public.dhe.ibm.com/software/uk/itsolutions/businessconnect2013/dk_pdf/SmarterBusDenmark_BigData_MDoylev2external.pdf (2013)
Barbierato, E., Gribaudo, M., Iacono, L.: Performance evaluation of nosql big-data applications using multi-formalism models. Future Gener. Comput. Syst. 37, 345–353 (2014)
Zhang, C., Liu, X.: Hbasemq: a distributed message queuing system on clouds with hbase. In: Proceedings IEEE, INFOCOM, pp. 40–44 (2013)
Yang, C.T., Liao, C.J., Liu, J.C., Den, W., Chou, Y.C., Tsai, J.J.: Construction and application of an intelligent air quality monitoring system for healthcare environment. J. Med. Syst. 38, 15 (2014)
Yang, C.-T., Shih, W.-C., Chen, L.-T., Kuo, C.-T., Jiang, F.-C., Leu, F.-Y.: Accessing medical image file with co-allocation hdfs in cloud. Future Gener. Comput. Syst. 43–44, 61–73 (2015)
Gu, L., Li, H.: Memory or time: Performance evaluation for iterative operation on hadoop and spark. In: IEEE 10th International Conference on, High Performance Computing and Communications 2013 IEEE International Conference on Embedded and Ubiquitous Computing (HPCC_EUC), pp. 721–727 (2013)
Urbani, J., Margara, A., Jacobs, C., Voulgaris, S., Bal, H.: Ajira: A lightweight distributed middleware for mapreduce and stream processing. In: IEEE 34th International Conference on, Distributed Computing Systems (ICDCS), pp. 545–554 (2014)
Zhang, J., You, S., Gruenwald, L.: High-performance spatial query processing on big taxi trip data using gpgpus. In: IEEE International Congress on, Big Data (BigData Congress), pp. 72–79 (2014)
Wang, L., Hu, S.W., Liu, P.: A computing perspective on smart city. IEEE Trans. Comput. 65, 1337–1338 (2016)
Dobre, C., Xhafa, F.: Intelligent services for big data science. Future Gener. Comput. Syst. 37, 267–281 (2014)
Zeng, Y., Lan, J., Ran, B., Jiang, Y.: A novel multisensor traffic state assessment system based on incomplete data. ScientificWorld J. 2014, 532602 (2014)
Jin, Y., Deyu, T., Yi, Z.: A distributed storage model for ehr based on hbase. In: International Conference on, Information Management, Innovation Management and Industrial Engineering (ICIII), vol. 2, pp. 369–372 (2011)
Ding, H., Jin, Y., Cui, Y., Yang, T.; Distributed storage of network measurement data on hbase. In: IEEE 2nd International Conference on, Cloud Computing and Intelligent Systems (CCIS), vol. 02, pp. 716–720 (2012)
Vora, M.: Hadoop-hbase for large-scale data. In: International Conference on, Computer Science and Network Technology (ICCSNT), vol. 1, pp. 601–605 (2011)
The hadoop distributed file system: Architecture and design. http://hadoop.apache.org/docs/r0.18.0/hdfs_design.pdf (2007)
Apache spark. https://spark.apache.org/ (2015)
Campello, R., Moulavi, D., Sander, J.: Density-based clustering based on hierarchical density estimates. In: Pei, J., Tseng, V., Cao, L., Motoda, H., Xu, G. (eds.) Advances in Knowledge Discovery and Data Mining. Lecture Notes in Computer Science, vol. 7819, pp. 160–172. Springer, Berlin (2013)
MacQueen, J.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability Statistics, vol. 1, pp. 281–297. University of California Press, Berkeley, (1967)
Frahling, G., Sohler, C.: A fast k-means implementation using coresets. In: Proceedings of the 22nd ACM Symposium on Computational Geometry, pp. 135–143. Sedona, Arizona, USA, June 5–7, (2006)
Fuzzy clustering. http://en.wikipedia.org/wiki/Fuzzy_clustering (2015)
Scala of fuzzy-c-means clustering. http://gist.github.com/kralo/8721440 (2015)
Acknowledgements
This work was supported in part by the Ministry of Science and Technology, Taiwan, under Grants numbers 104-2221-E-029-010-MY3, 103-2632-H-029-001-MY2, and 105-2634-E-029-001.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yang, CT., Chen, ST. & Yan, YZ. The implementation of a cloud city traffic state assessment system using a novel big data architecture. Cluster Comput 20, 1101–1121 (2017). https://doi.org/10.1007/s10586-017-0846-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10586-017-0846-z