skip to main content
10.1145/3409390.3409394acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicppConference Proceedingsconference-collections
research-article

Devise Sparse Compression Schedulers to Enhance FastText Methods

Published: 17 August 2020 Publication History

Abstract

In natural language processing(NLP), the general way to understand the meaning of a word is via word embedding. The word embedding training model can convert words into multidimensional vectors and make the words that do not know “meaning” into vectors with “meaning”. Famous word embedding training models, include models such as FastText, Word2Vec, and GloVe. They can train words into vectors and then they are used for further semantic classifications. In this paper, we work on the efficient support for the FastText. FastText is an open source library created by Facebook(FAIR) lab that allows users to learn word embedding and text classification. We focus on the word representation application in FastText, in which general matrix-Vector multiplication(GEMV) is one of the most computationally intensive operations. In this paper, we adjust the software architecture of FastText, and pre-process the pre-trained model offline. In addition, we introduce a new accelerating method with sparse matrix compression in Halide, which improves performance by compressing the matrix. Our support with Halide sparse compression schedulers include hybrid compression schemes and re-ordering methods to improve the performance.

References

[1]
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5 (2017), 135–146.
[2]
Rong-Guey Chang, Tyng-Ruey Chuang, and Jenq Kuen Lee. 2001. Parallel sparse supports for array intrinsic functions of Fortran 90. The Journal of supercomputing 18, 3 (2001), 305–339.
[3]
Rong-Guey Chang, Tyng-Ruey Chuang, and Jenq Kuen Lee. 2004. Support and optimization for parallel sparse programs with array intrinsics of Fortran 90. Parallel Comput. 30, 4 (2004), 527–550.
[4]
Changwan Hong, Aravind Sukumaran-Rajam, Bortik Bandyopadhyay, Jinsung Kim, Süreyya Emre Kurt, Israt Nisa, Shivani Sabhlok, Ümit V. Çatalyürek, Srinivasan Parthasarathy, and P. Sadayappan. 2018. Efficient sparse-matrix multi-vector product on GPUs. In HPDC ’18.
[5]
Chia-Chen Hsu, Cheng-Yen Lin, Shin Kai Chen, Chih-Wei Liu, and Jenq-Kuen Lee. 2014. Optimized Memory Access Support for Data Layout Conversion on Heterogeneous Multi-core Systems. WOS:000358220900016 (2014). https://ir.nctu.edu.tw/handle/11536/128539
[6]
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. FastText.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651(2016).
[7]
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. Association for Computational Linguistics, 427–431.
[8]
Chao-Lin Lee, Chen-Ting Chao, Jenq-Kuen Lee, Chung-Wen Huang, and Ming-Yu Hung. 2019. Sparse-Matrix Compression Primitives with OpenCL Framework to Support Halide. In Proceedings of the International Workshop on OpenCL. 1–2.
[9]
Chao-Lin Lee, Chen-Ting Chao, Jenq-Kuen Lee, Ming-Yu Hung, and Chung-Wen Huang. 2019. Accelerate DNN Performance with Sparse Matrix Compression in Halide. In Proceedings of the 48th International Conference on Parallel Processing: Workshops (Kyoto, Japan) (ICPP 2019). Association for Computing Machinery, New York, NY, USA, Article 14, 6 pages. https://doi.org/10.1145/3339186.3339194
[10]
Jiajia Li, Bora Uçar, Umit Catalyurek, J. Sun, Kevin Barker, and Richard Vuduc. 2019. Efficient and effective sparse tensor reordering. 227–237. https://doi.org/10.1145/3330345.3330366
[11]
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Portland, Oregon, USA, 142–150. http://www.aclweb.org/anthology/P11-1015
[12]
Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. CoRR abs/1301.3781(2013).
[13]
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, 1532–1543. https://doi.org/10.3115/v1/D14-1162
[14]
Jonathan Ragan-Kelley, Andrew Adams, Sylvain Paris, Marc Levoy, Saman Amarasinghe, and Frédo Durand. 2012. Decoupling algorithms from schedules for easy optimization of image processing pipelines. (2012).
[15]
Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. ACM SIGPLAN Notices 48, 6 (2013), 519–530.
[16]
Rong-Guey Chang, Jia-Shin Li, Jenq Kuen Lee, and Tyng-Ruey Chuang. 2001. Probabilistic inference schemes for sparsity structures of Fortran 90 array intrinsics. In International Conference on Parallel Processing, 2001.61–68.
[17]
W. F. Tinney and J. W. Walker. 1967. Direct solutions of sparse network equations by optimally ordered triangular factorization. Proc. IEEE 55, 11 (1967), 1801–1809.

Cited By

View all
  • (2022)CBVoSD: context based vectors over sentiment domain ensemble model for review classificationThe Journal of Supercomputing10.1007/s11227-021-04132-578:5(6411-6447)Online publication date: 1-Apr-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICPP Workshops '20: Workshop Proceedings of the 49th International Conference on Parallel Processing
August 2020
186 pages
ISBN:9781450388689
DOI:10.1145/3409390
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. NLP
  2. neural networks
  3. sparse compression
  4. word embedding
  5. word presentation

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICPP Workshops '20
ICPP Workshops '20: Workshops
August 17 - 20, 2020
AB, Edmonton, Canada

Acceptance Rates

Overall Acceptance Rate 91 of 313 submissions, 29%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2022)CBVoSD: context based vectors over sentiment domain ensemble model for review classificationThe Journal of Supercomputing10.1007/s11227-021-04132-578:5(6411-6447)Online publication date: 1-Apr-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media