Skip to main content
SpringerLink
Log in

High-Performance and Customizable Vector Retrieval Service Based on Faiss in Power Grid Scenarios

  • Conference paper
  • First Online:
Smart Computing and Communication (SmartCom 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13202))

Included in the following conference series:

  • 1103 Accesses

Abstract

With the rapid development of machine learning and deep learning, more and more services in the power grid have introduced machine learning and deep learning technologies, and related application scenarios and services have become more and more, especially vector retrieval services. With the increase of the amount of data and the increase of the demand, higher requirements are put forward for the vector retrieval service performance and service management. In order to solve this problem, this paper designs a high-performance and customizable vector retrieval service, referred to as HCFRS. The HCVRS service uses Fiass as the underlying framework of the vector retrieval service, supporting functions such as service registration, service unloading, resource allocation, load balancing, and data management. This paper verifies whether the HVCRS service meets the service design requirements from the three aspects of functional testing, accuracy testing and performance testing. The experimental results show that the HVCRS service has complete functions and good performance, which basically solves the difficulties encountered by the State Grid in vector retrieval services.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Witten, I.H., et al.: Data Mining: Practical Machine Learning Tools and Techniques, 4th edn. Morgan Kaufmann, Cambridge (2017)

    MATH  Google Scholar 

  2. Marsland, S.: Machine Learning: An Algorithmic Perspective, 2 edn. CRC Press, Boca Raton (2014, 2015)

    Google Scholar 

  3. Kang, M., Choi, E.: Machine Learning: Concepts, Tools and Data Visualization. World Scientific Publishing Co., Pte. Ltd., Singapore (2021)

    Book  Google Scholar 

  4. Bhattacharyya, S., et al.: Deep Learning: Research and Applications. De Gruyter, Berlin (2020)

    Book  Google Scholar 

  5. Ramsundar, B., et al.: Deep Learning for the Life Sciences: Applying Deep Learning to Genomics, Microscopy, Drug Discovery and More, 1st edn. O’Reilly Media, Sebastopol (2019)

    Google Scholar 

  6. Silhavy, P., Silhavy, R., Prokopova, Z.: Intelligent Systems Applications in Software Engineering. In: Proceedings of 3rd Computational Methods in Systems and Software, 20191046 (2019)

    Google Scholar 

  7. Hörmann, B.O., Bizubac, D., Popa, M.S.: Industrial intelligent software applications of the overall equipment effectiveness in manufacturing. In: MATEC Web of Conference, vol. 299, p. 5011 (2019)

    Google Scholar 

  8. Qiuyi, L.: Intelligent terminal multi-application software control method (2020)

    Google Scholar 

  9. Kim, N., et al.: Dynamic patterns of industry convergence: evidence from a large amount of unstructured data. Res. Policy 44(9), 1734–1748 (2015)

    Article  Google Scholar 

  10. Baars, H., Kemper, H.: Management support with structured and unstructured data-an integrated business intelligence framework. Inf. Syst. Manag. 25(2), 132–148 (2008)

    Article  Google Scholar 

  11. Han, J., Kamber, M., Pei, J.: Data Mining: Concepts and Techniques, 3rd edn. Elsevier, Burlington (2012)

    MATH  Google Scholar 

  12. Torgo, L.: Data Mining with R: Learning with Case Studies, 2nd edn. CRC Press, Taylor & Francis Group, Boca Raton (2017)

    Google Scholar 

  13. Li, G., et al.: Recommendation system based on users preference mining generative adversarial networks. Jisuanji Kexue Yu Tansuo 14(5), 803–814 (2020)

    Google Scholar 

  14. Chen, R., Hendry, Liu, L.: Enhancing personal recommendation system using familiarity factor on social network (2017)

    Google Scholar 

  15. Wu, S.Y.J.: Recommendation system for medical consultation integrating knowledge graph and deep learning methods. Jisuanji Kexue Yu Tansuo, 15(8), 1432–1440 (2021)

    Google Scholar 

  16. Yunfei, Z., Yeli, L., Huayan, S.: Research of personalized recommendation system based on deep neural network. Diànzǐ Jìshù Yīngyòng 45(1), 14–18 (2019)

    Google Scholar 

  17. Wang, S., Lo, D., Vasilescu, B., Serebrenik, A.: EnTagRec ++: an enhanced tag recommendation system for software information sites. Empirical Softw. Eng. 23(2), 800–832 (2017). https://doi.org/10.1007/s10664-017-9533-1

    Article  Google Scholar 

  18. Qiu, H., Qiu, M., Lu, Z.: Selective encryption on ECG data in body sensor network based on supervised machine learning. Inf. Fus. 55, 59–67 (2020)

    Article  Google Scholar 

  19. Qiu, L., Gai, K., Qiu, M.: Optimal big data sharing approach for tele-health in cloud computing. In: IEEE SmartCloud, pp. 184–189 (2016)

    Google Scholar 

  20. Lu, Z., Wang, N., Wu, J., Qiu, M.: IoTDeM: an IoT big data-oriented MapReduce performance prediction extended model in multiple edge clouds. J. Parallel Distrib. Comput. 118, 316–327 (2018)

    Article  Google Scholar 

  21. Qiu, M., Chen, Z., Liu, M.: Low-power low-latency data allocation for hybrid scratch-pad memory. IEEE Embedd. Syst. Lett. 6(4), 69–72 (2014)

    Article  Google Scholar 

  22. Zhang, L., Qiu, M., Tseng, W., Sha, E.: Variable partitioning and scheduling for MPSoC with virtually shared scratch pad memory. J. Signal Process. Syst. 58(2), 247–265 (2010). https://doi.org/10.1007/s11265-009-0362-3

    Article  Google Scholar 

  23. Qiu, M., Xue, C., Shao, Z., Sha, E.: Energy minimization with soft real-time and DVS for uniprocessor and multiprocessor embedded systems. In: IEEE DATE Conference, pp. 1–6 (2007)

    Google Scholar 

  24. Lu, R., Jin, X., Zhang, S., Qiu, M., Wu, X.: A study on big knowledge and its engineering issues. IEEE Trans. Knowl. Data Eng. 31(9), 1630–1644 (2018)

    Article  Google Scholar 

  25. Qiu, M., Cao, D., Su, H., Gai, K.: Data transfer minimization for financial derivative pricing using Monte Carlo simulation with GPU in 5G. Int’l J. Comm. Sys. 29(16), 2364–2374 (2016)

    Article  Google Scholar 

  26. Liu, M., Zhang, S., et al.: H infinite state estimation for discrete-time chaotic systems based on a unified model. IEEE Trans. Syst. Man Cybern. (B) (2012)

    Google Scholar 

  27. Mikolov, T., et al.: Efficient estimation of word representations in vector space (2013)

    Google Scholar 

  28. Barkan, O., Koenigstein, N.: ITEM2VEC: neural item embedding for collaborative filtering. (2016). https://doi.org/10.1109/MLSP.2016.7738886

  29. Perozzi, B., Al-Rfou, R., Skiena, S.: DeepWalk: online learning of social representations (2014). https://doi.org/10.1145/2623330.2623732

  30. Huifu, Z., Kazhong, D., Hongdong, F.: SAR images unsupervised change detection based on combination of texture feature vector with maximum entropy principle. Ce Hui Xue Bao 45(3), 339–346 (2016)

    Google Scholar 

  31. Lakshmi Priya, G.G., Domnic, S.: Walsh-Hadamard Transform Kernel-Based Feature Vector for Shot Boundary Detection. IEEE Trans. Image Process. 23(12), 5187–5197 (2014)

    Article  MathSciNet  Google Scholar 

  32. Hu, S., et al.: Multi-dimensional big data feature attribute processing method and device, terminal and storage medium (2020)

    Google Scholar 

  33. Ahmadi, N., Akbarizadeh, G.: Hybrid robust iris recognition approach using iris image pre-processing, two-dimensional gabor features and multi-layer perceptron neural network/PSO. IET Biometrics 7(2), 153–162 (2018)

    Article  Google Scholar 

  34. Heimrath, K., et al.: Modulation of pre-attentive spectro-temporal feature processing in the human auditory system by HD-tDCS. Eur. J. Neurosci. 41(12), 1580–1586 (2015)

    Article  Google Scholar 

  35. Zhichang, Y., Xu, Z., Yingfeng, Z.: Distributed vector retrieval engine (2019)

    Google Scholar 

  36. Shishi, W., Xi, Z., Zhiqiang, Y.: Vector retrieval method, system, device and medium (2020)

    Google Scholar 

  37. Bettenhausen, M.H., et al.: A nonlinear optimization algorithm for WindSat wind vector retrievals. IEEE Trans. Geosci. Remote Sens. 44(3), 597–610 (2006)

    Article  Google Scholar 

  38. Zhimin, L., Gang, T.: Conversation response method, device and system based on vector retrieval (2020)

    Google Scholar 

  39. Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs (2017)

    Google Scholar 

  40. Liu, Y., et al.: SK-LSH: an efficient index structure for approximate nearest neighbor search. Proc. VLDB Endow. 7(9), 745–756 (2014)

    Article  Google Scholar 

  41. Dingcheng, M., Xiaoliang, X., Yuxiang. W.: Approximate nearest neighbor search method combining VP tree and guide nearest neighbor graph (2021)

    Google Scholar 

  42. Komorowski, M., Trzciński, T.: Random binary search trees for approximate nearest neighbour search in binary spaces. Appl. Soft Comput. 79, 87–93 (2019)

    Article  Google Scholar 

  43. Friston, K.J., et al.: Functional connectivity: the principal-component analysis of large (PET) data sets. J. Cereb. Blood Flow Metab. 13(1), 5–14 (1993, 2016)

    Google Scholar 

  44. Abdi, H., Williams, L.J.: Principal component analysis. Wiley Interdisc. Rev. Comput. Stat. 2(4), 433–459 (2010)

    Article  Google Scholar 

  45. Jégou, H., Douze, M., Schmid, C.: Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. 33(1), 117–128 (2011)

    Article  Google Scholar 

  46. André, F., Kermarrec, A., Le Scouarnec, N.: Cache locality is not enough: high-performance nearest neighbor search with product quantization fast scan. Proc. VLDB 9(4), 288–299 (2015)

    Article  Google Scholar 

  47. Goswami, P., et al.: An efficient multi-resolution framework for high quality interactive rendering of massive point clouds using multi-way KD-trees. Vis. Comput. 29(1), 69–83 (2013). https://doi.org/10.1007/s00371-012-0675-2

    Article  Google Scholar 

  48. Adams, A., et al.: Gaussian KD-trees for fast high-dimensional filtering. ACM Trans. Graph. 28(3), 1–12 (2009)

    Article  MathSciNet  Google Scholar 

  49. Keil, J.M., Vassilev, T.S.: The relative neighbourhood graph is a part of every 30° -triangulation. Inf. Process. Lett. 109(2), 93–97 (2008)

    Article  MathSciNet  Google Scholar 

  50. Kleindessner, M., Von Luxburg, U.: Lens depth function and k-relative neighborhood graph: versatile tools for ordinal data analysis. J. Mach. Learn. Res. 18, 1–52 (2017)

    MathSciNet  MATH  Google Scholar 

  51. Melchert, O.: Percolation thresholds on planar Euclidean relative-neighborhood graphs. Phys. Rev. E, Stat. Nonlinear Soft Matter Phys. 87(4), 042106 (2013)

    Article  Google Scholar 

  52. Keil, J.M., Vassilev, T.S.: The relative neighbourhood graph is a part of every 30°-triangulation. Inf. Process. Lett. 109(2), 93–97 (2008)

    Article  MathSciNet  Google Scholar 

  53. Kunz, P., et al.: Most classic problems remain np-hard on relative neighborhood graphs and their relatives (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Big Data Center of State Grid Corporation of China, Beijing, 100000, China

    Pengyu Zhang

Authors
  1. Pengyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Texas A&M University-Commerce, Commerce, TX, USA

    Meikang Qiu

  2. Beijing Institute of Technology, Beijing, Beijing, China

    Keke Gai

  3. Tsinghua University, Beijing, China

    Han Qiu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, P. (2022). High-Performance and Customizable Vector Retrieval Service Based on Faiss in Power Grid Scenarios. In: Qiu, M., Gai, K., Qiu, H. (eds) Smart Computing and Communication. SmartCom 2021. Lecture Notes in Computer Science, vol 13202. Springer, Cham. https://doi.org/10.1007/978-3-030-97774-0_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-97774-0_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-97773-3

  • Online ISBN: 978-3-030-97774-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation