Skip to main content
SpringerLink
Log in

Fast Training of POI Recommendation Models Using Gradient Compression

  • Conference paper
  • First Online:
Book cover Spatial Data and Intelligence (SpatialDI 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12567))

Included in the following conference series:

  • 733 Accesses

Abstract

Point-of-Interest (POI) recommendation plays an important role in location-based services. Regarding POI recommendation as a trajectory prediction problem, attention-based models have shown promising achievements to tackle this problem. With the rapid development of LBSN applications, modern training datasets and POI recommendation models are becoming increasingly large and complicated, which results in time-consuming training procedures. Existing solutions adopt data parallel training for POI models, but they still suffer from the high latency of inter-device data communication. To address this issue, we propose a fast training method for POI recommendation models. It conducts a layer-wise top-k gradient compression to reduce the overhead of data transfer. Meanwhile, it modifies the Adam optimizer with an error feedback mechanism to avoid the decrease of POI recommendation accuracy resulting from gradient compression. Experimental results show that our methods can achieve similar even better accuracy than uncompressed methods while the communication time is shorter.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://foursquare.com/about.

References

  1. Aji, A.F., Heafield, K.: Sparse communication for distributed gradient descent. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, 9–11 September 2017, pp. 440–445 (2017)

    Google Scholar 

  2. Alistarh, D., Grubic, D., Li, J., Tomioka, R., Vojnovic, M.: QSGD: communication-efficient SGD via gradient quantization and encoding. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA, pp. 1709–1720 (2017)

    Google Scholar 

  3. Alistarh, D., Hoefler, T., Johansson, M., Konstantinov, N., Khirirat, S., Renggli, C.: The convergence of sparsified gradient methods. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018, Montréal, Canada, pp. 5977–5987 (2018)

    Google Scholar 

  4. Bernstein, J., Wang, Y., Azizzadenesheli, K., Anandkumar, A.: SIGNSGD: compressed optimisation for non-convex problems. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, 10–15 July 2018, pp. 559–568 (2018)

    Google Scholar 

  5. Chang, B., Park, Y., Park, D., Kim, S., Kang, J.: Content-aware hierarchical point-of-interest embedding model for successive POI recommendation. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, 13–19 July 2018, Stockholm, Sweden, pp. 3301–3307 (2018)

    Google Scholar 

  6. Chen, C., Choi, J., Brand, D., Agrawal, A., Zhang, W., Gopalakrishnan, K.: Adacomp: adaptive residual gradient compression for data-parallel distributed training. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, 2–7 February 2018, pp. 2827–2835 (2018)

    Google Scholar 

  7. Cheng, C., Yang, H., King, I., Lyu, M.R.: Fused matrix factorization with geographical and social influence in location-based social networks. In: Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, 22–26 July 2012, Toronto, Ontario, Canada (2012)

    Google Scholar 

  8. Cheng, C., Yang, H., Lyu, M.R., King, I.: Where you like to go next: successive point-of-interest recommendation. In: IJCAI 2013, Proceedings of the 23rd International Joint Conference on Artificial Intelligence, Beijing, China, 3–9 August 2013, pp. 2605–2611 (2013)

    Google Scholar 

  9. Cheng, W., Shen, Y., Zhu, Y., Huang, L.: DELF: a dual-embedding based deep latent factor model for recommendation. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, 13–19 July 2018, Stockholm, Sweden, pp. 3329–3335 (2018)

    Google Scholar 

  10. Feng, J., et al.: DeepMove: predicting human mobility with attentional recurrent networks. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, 23–27 April 2018, pp. 1459–1468 (2018)

    Google Scholar 

  11. Feng, S., Cong, G., An, B., Chee, Y.M.: PoI2VEC: geographical latent representation for predicting future visitors. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 4–9 February 2017, San Francisco, California, USA, pp. 102–108 (2017)

    Google Scholar 

  12. Feng, S., Li, X., Zeng, Y., Cong, G., Chee, Y.M., Yuan, Q.: Personalized ranking metric embedding for next new POI recommendation. In: Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, 25–31 July 2015, pp. 2069–2075 (2015)

    Google Scholar 

  13. Gao, H., Tang, J., Hu, X., Liu, H.: Content-aware point of interest recommendation on location-based social networks. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 25–30 January 2015, Austin, Texas, USA, pp. 1721–1727 (2015)

    Google Scholar 

  14. He, J., Li, X., Liao, L., Song, D., Cheung, W.K.: Inferring a personalized next point-of-interest recommendation model with latent behavior patterns. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 12–17 February 2016, Phoenix, Arizona, USA, pp. 137–143 (2016)

    Google Scholar 

  15. Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016. Conference Track Proceedings (2016)

    Google Scholar 

  16. Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D.P., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. In: Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, 6–10 August 2018, pp. 876–885 (2018)

    Google Scholar 

  17. Karimireddy, S.P., Rebjock, Q., Stich, S.U., Jaggi, M.: Error feedback fixes sign SGD and other gradient compression schemes. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019, Long Beach, California, USA, pp. 3252–3261 (2019)

    Google Scholar 

  18. Li, R., Shen, Y., Zhu, Y.: Next point-of-interest recommendation with temporal and multi-level context attention. In: IEEE International Conference on Data Mining, ICDM 2018, Singapore, 17–20 November 2018, pp. 1110–1115 (2018)

    Google Scholar 

  19. Lian, D., Wu, Y., Ge, Y., Xie, X., Chen, E.: Geography-aware sequential location recommendation. In: KDD 2020: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, 23–27 August 2020, pp. 2009–2019 (2020)

    Google Scholar 

  20. Lian, D., Zhao, C., Xie, X., Sun, G., Chen, E., Rui, Y.: Geomf: joint geographical modeling and matrix factorization for point-of-interest recommendation. In: The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2014, New York, NY, USA, 24–27 August 2014, pp. 831–840 (2014)

    Google Scholar 

  21. Lian, D., Zheng, V.W., Xie, X.: Collaborative filtering meets next check-in location prediction. In: 22nd International World Wide Web Conference, WWW 2013, Rio de Janeiro, Brazil, 13–17 May 2013, Companion Volume, pp. 231–232 (2013)

    Google Scholar 

  22. Lin, Y., Han, S., Mao, H., Wang, Y., Dally, B.: Deep gradient compression: Reducing the communication bandwidth for distributed training. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3 2018, Conference Track Proceedings (2018)

    Google Scholar 

  23. Liu, Q., Wu, S., Wang, L., Tan, T.: Predicting the next location: a recurrent model with spatial and temporal contexts. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 12–17 February 2016, Phoenix, Arizona, USA, pp. 194–200 (2016)

    Google Scholar 

  24. Liu, X., Li, Y., Tang, J., Yan, M.: A double residual compression algorithm for efficient distributed learning. In: The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26–28 August 2020, Palermo, Sicily, Italy, pp. 133–143 (2020)

    Google Scholar 

  25. Manotumruksa, J., Macdonald, C., Ounis, I.: A contextual attention recurrent architecture for context-aware venue recommendation. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, 08–12 July 2018, pp. 555–564 (2018)

    Google Scholar 

  26. Seide, F., Fu, H., Droppo, J., Li, G., Yu, D.: 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In: INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, 14–18 September 2014, pp. 1058–1062 (2014)

    Google Scholar 

  27. Stich, S.U., Cordonnier, J., Jaggi, M.: Sparsified SGD with memory. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018, Montréal, Canada, pp. 4452–4463 (2018)

    Google Scholar 

  28. Strom, N.: Scalable distributed DNN training using commodity GPU cloud computing. In: INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, 6–10 September 2015, pp. 1488–1492 (2015)

    Google Scholar 

  29. Sun, K., Qian, T., Chen, T., Liang, Y., Nguyen, Q.V.H., Yin, H.: Where to go next: Modeling long- and short-term user preferences for point-of-interest recommendation. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, 7–12 February 2020, pp. 214–221 (2020)

    Google Scholar 

  30. Tang, H., Yu, C., Lian, X., Zhang, T., Liu, J.: Doublesqueeze: parallel stochastic gradient descent with double-pass error-compensated compression. In: International Conference on Machine Learning, pp. 6155–6165. PMLR (2019)

    Google Scholar 

  31. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA, pp. 5998–6008 (2017)

    Google Scholar 

  32. Vogels, T., Karimireddy, S.P., Jaggi, M.: PowerSGD: practical low-rank gradient compression for distributed optimization. In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8–14 December 2019, Vancouver, BC, Canada, pp. 14236–14245 (2019)

    Google Scholar 

  33. Wangni, J., Wang, J., Liu, J., Zhang, T.: Gradient sparsification for communication-efficient distributed optimization. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3–8 December 2018, Montréal, Canada, pp. 1306–1316 (2018)

    Google Scholar 

  34. Wen, W., et al.: Terngrad: ternary gradients to reduce communication in distributed deep learning. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach, CA, USA, pp. 1509–1519 (2017)

    Google Scholar 

  35. Ye, M., Yin, P., Lee, W., Lee, D.L.: Exploiting geographical influence for collaborative point-of-interest recommendation. In: Proceeding of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2011, Beijing, China, 25–29 July 2011, pp. 325–334 (2011)

    Google Scholar 

  36. Yin, H., Wang, W., Wang, H., Chen, L., Zhou, X.: Spatial-aware hierarchical collaborative deep learning for POI recommendation. IEEE Trans. Knowl. Data Eng. 29(11), 2537–2551 (2017)

    Article  Google Scholar 

  37. Zhao, P., et al.: Where to go next: a spatio-temporal gated network for next POI recommendation. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, 27 January - 1 February 2019, pp. 5877–5884 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Science and Technology of China, Hefei, Anhui, 230026, China

    Hao Sun, Yunzhuo Wang, Jingwei Sun & Guangzhong Sun

Authors
  1. Hao Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yunzhuo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jingwei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guangzhong Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingwei Sun .

Editor information

Editors and Affiliations

  1. Renmin University of China, Beijing, China

    Xiaofeng Meng

  2. Microsoft Research Asia, Beijing, China

    Xing Xie

  3. Shenzhen University, Shenzhen, China

    Yang Yue

  4. Chinese Academy of Sciences, Beijing, China

    Zhiming Ding

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sun, H., Wang, Y., Sun, J., Sun, G. (2021). Fast Training of POI Recommendation Models Using Gradient Compression. In: Meng, X., Xie, X., Yue, Y., Ding, Z. (eds) Spatial Data and Intelligence. SpatialDI 2020. Lecture Notes in Computer Science(), vol 12567. Springer, Cham. https://doi.org/10.1007/978-3-030-69873-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-69873-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-69872-0

  • Online ISBN: 978-3-030-69873-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation