Skip to main content
SpringerLink
Log in

Hard Disk Failure Prediction via Transfer Learning

  • Conference paper
  • First Online:
Big Data and Security (ICBDS 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1415))

Included in the following conference series:

  • 988 Accesses

Abstract

Due to the large-scale growth of data, the storage scale of data centers is getting larger and larger. Hard disk is the main storage medium, once a failure occurs, it will bring huge losses to users and enterprises. In order to improve the reliability of storage systems, many machine learning methods have been widely employed to predict hard disk failure in the past few decades. However, due to the large number of different models of hard disks in the heterogeneous disk system, traditional machine learning methods cannot build a general model. Inspired by a DANN based unsupervised domain adaptation approach for image classification, in this paper, we propose the DFPTL (Disk Failure Prediction via Transfer Learning) approach, which introduce the DANN approach to predict failure in heterogeneous disk systems by reducing the distribution differences between different models of disk datasets. This approach only needs unlabeled data (the target domain) of a specific disk model and the labeled data (the source domain) collected from a different disk model from the same manufacturer. Experimental results on real-world datasets demonstrate that DFPTL can achieve adaptation effect in the presence of domain shifts and outperform traditional machine learning algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Vishwanath, K.V., Nagappan, N.: Characterizing cloud computing hardware reliability. In: Proceedings of the 1st ACM symposium on Cloud computing, pp. 193–204 (2010)

    Google Scholar 

  2. Allen, B.: Monitoring hard disks with smart. Linux J. 117, 74–77 (2004)

    Google Scholar 

  3. Eckart, B., Chen, X., He X., et al.: Failure prediction models for proactive fault tolerance within storage systems. In: 2008 IEEE International Symposium on Modeling, Analysis and Simulation of Computers and Telecommunication Systems, pp. 1–8. IEEE (2008)

    Google Scholar 

  4. Murray, J.F., Hughes, G.F., Kreutz-Delgado, K.: Machine learning methods for predicting failures in hard drives: a multiple-instance application. J. Mach. Learn. Res. 6(May), 783–816 (2005)

    MathSciNet  MATH  Google Scholar 

  5. Xiao, J., Xiong, Z., Wu, S., et al.: Disk failure prediction in data centers via online learning. In: Proceedings of the 47th International Conference on Parallel Processing, pp. 1–10 (2018)

    Google Scholar 

  6. Li, J., Ji, X., Jia, Y., et al.: Hard drive failure prediction using classification and regression trees. In: 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 383–394. IEEE (2014)

    Google Scholar 

  7. Murray, J.F., Hughes, G.F., Kreutz-Delgado, K.: Hard drive failure prediction using non-parametric statistical methods. In: Proceedings of ICANN/ICONIP (2003)

    Google Scholar 

  8. Yang, W., Hu, D., Liu, Y., et al.: Hard drive failure prediction using big data. In: 2015 IEEE 34th Symposium on Reliable Distributed Systems Workshop (SRDSW), pp. 13–18. IEEE (2015)

    Google Scholar 

  9. Zhao, Y., Liu, X., Gan, S., et al.: Predicting disk failures with HMM-and HSMM-based approaches. In: Industrial Conference on Data Mining, pp. 390–404. Springer, Berlin, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14400-4_30

  10. Botezatu, M.M., Giurgiu, I., Bogojeska, J., et al.: Predicting disk replacement towards reliable data centers. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 39–48 (2016)

    Google Scholar 

  11. Zhu, B., Wang, G., Liu, X., et al.: Proactive drive failure prediction for large scale storage systems. In: 2013 IEEE 29th symposium on mass storage systems and technologies (MSST), pp. 1–5. IEEE (2013)

    Google Scholar 

  12. Xu, Y., Sui, K., Yao, R., et al.: Improving service availability of cloud systems by predicting disk error. In: 2018 {USENIX} Annual Technical Conference ({USENIX}{ATC} 18), pp. 481–494 (2018)

    Google Scholar 

  13. Pereira, F.L.F., dos Santos Lima, F D., de Moura Leite, L G., et al.: Transfer learning for Bayesian networks with application on hard disk drives failure prediction. In: 2017 Brazilian Conference on Intelligent Systems (BRACIS), pp. 228–233. IEEE (2017)

    Google Scholar 

  14. Jiang, W., Hu, C., Zhou, Y., et al.: Are disks the dominant contributor for storage failures? a comprehensive study of storage subsystem failure characteristics. ACM Trans. Storage (TOS). 4(3), 1–25 (2008)

    Article  Google Scholar 

  15. Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International conference on machine learning, pp. 1180–1189 (2015)

    Google Scholar 

  16. Hughes, G.F., Murray, J.F., Kreutz-Delgado, K., et al.: Improved disk-drive failure warnings. IEEE Trans. Reliab. 51(3), 350–357 (2002)

    Article  Google Scholar 

  17. Hamerly, G., Elkan, C.: Bayesian approaches to failure prediction for disk drives. ICML 1, 202–209 (2001)

    Google Scholar 

  18. Wang, Y., Ma, E.W.M., Chow, T.W.S., et al.: A two-step parametric method for failure prediction in hard disk drives. IEEE Trans. Ind. Inf. 10(1), 419–430 (2013)

    Article  Google Scholar 

  19. Xu, C., Wang, G., Liu, X., et al.: Health status assessment and failure prediction for hard drives with recurrent neural networks. IEEE Trans. Comput. 65(11), 3502–3508 (2016)

    Article  MathSciNet  Google Scholar 

  20. Zhu, Y., Zhuang, F., Wang, J., et al.: Multi-representation adaptation network for cross-domain image classification. Neural Netw. 119, 214–221 (2019)

    Article  Google Scholar 

  21. Prettenhofer, P., Stein, B.: Cross-language text classification using structural correspondence learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics, pp. 1118–1127 (2010)

    Google Scholar 

  22. Wang, J., Zheng, V.W., Chen, Y., et al.: Deep transfer learning for cross-domain activity recognition. In: proceedings of the 3rd International Conference on Crowd Science and Engineering, pp. 1–8 (2018)

    Google Scholar 

  23. Kermany, D.S., Goldbaum, M., Cai, W., et al.: Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5), 1122–1131 (2018)

    Article  Google Scholar 

  24. Zhang, J., Zhou, K., Huang, P., et al.: Transfer learning based failure prediction for minority disks in large data centers of heterogeneous disk systems. In: Proceedings of the 48th International Conference on Parallel Processing, pp.1–10 (2019)

    Google Scholar 

  25. Jiang, T., Zeng, J., Zhou, K., et al.: Lifelong disk failure prediction via GAN-Based anomaly detection. In: 2019 IEEE 37th International Conference on Computer Design (ICCD), pp. 199–207. IEEE (2019)

    Google Scholar 

  26. Zhang, X., Kim, J., Lin, Q., et al.: Cross-dataset time series anomaly detection for cloud systems. In: 2019 {USENIX} Annual Technical Conference ({USENIX}{ATC} 19), pp. 1063–1076 (2019)

    Google Scholar 

  27. Shimodaira, H.: Improving predictive inference under covariate shift by weighting the log-likelihood function. J. Stat. Plan. Infer. 90(2), 227–244 (2000)

    Article  MathSciNet  Google Scholar 

  28. Yang, Y., Loog, M.: Active learning using uncertainty information. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2646–2651. IEEE (2016)

    Google Scholar 

  29. Powers, D.M.: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation (2011)

    Google Scholar 

  30. He, H., Garcia, E.A.: Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. 21(9), 1263–1284 (2009)

    Article  Google Scholar 

  31. Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 413–422. IEEE (2008)

    Google Scholar 

  32. Breunig, M.M., Kriegel, H.P., Ng, R.T., et al.: LOF: identifying density-based local outliers. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pp. 93–104 (2000)

    Google Scholar 

  33. dos Santos Lima, F.D., Amaral, G.M.R., de Moura Leite, L.G., et al.: Predicting failures in hard drives with lstm networks. In: 2017 Brazilian Conference on Intelligent Systems (BRACIS), pp. 222–227. IEEE (2017)

    Google Scholar 

Download references

Acknowledgements

This research is supported by the key research and development program of Jiangsu province( BE2019012).

Author information

Authors and Affiliations

  1. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Rui Zhao, Donghai Guan, Hui Xiao, Weiwei Yuan & Yaofeng Tu

  2. Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing, China

    Rui Zhao, Donghai Guan, Hui Xiao & Weiwei Yuan

  3. Yanbian University, Yanbian, China

    Yuanfeng Jin

  4. Zayed University, Abu Dhabi, UAE

    Asad Masood Khattak

Authors
  1. Rui Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Donghai Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuanfeng Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hui Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weiwei Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yaofeng Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Asad Masood Khattak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Donghai Guan .

Editor information

Editors and Affiliations

  1. Nanjing Institute of Technology, Nanjing, China

    Yuan Tian

  2. Nanjing University of Information Science and Technology, Nanjing, China

    Tinghuai Ma

  3. King Saud Unviersity, Riyadh, Saudi Arabia

    Muhammad Khurram Khan

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhao, R. et al. (2021). Hard Disk Failure Prediction via Transfer Learning. In: Tian, Y., Ma, T., Khan, M.K. (eds) Big Data and Security. ICBDS 2020. Communications in Computer and Information Science, vol 1415. Springer, Singapore. https://doi.org/10.1007/978-981-16-3150-4_43

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-3150-4_43

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-3149-8

  • Online ISBN: 978-981-16-3150-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation