skip to main content
10.1145/3477495.3531761acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
short-paper

L3E-HD: A Framework Enabling Efficient Ensemble in High-Dimensional Space for Language Tasks

Published: 07 July 2022 Publication History

Abstract

Brain-inspired hyperdimensional computing (HDC) has been introduced as an alternative computing paradigm to achieve efficient and robust learning. HDC simulates cognitive tasks by mapping all data points to patterns of neural activity in the high-dimensional space, which has demonstrated promising performances in a wide range of applications such as robotics, biomedical signal processing, and genome sequencing. Language tasks, generally solved using machine learning methods, are widely deployed on low-power embedded devices. However, existing HDC solutions suffer from major challenges that impede the deployment of low-power embedded devices: the storage and computation overhead of HDC models grows dramatically with (i) the number of dimensions and (ii) the complex similarity metric during the inference.
In this paper, we proposed a novel ensemble framework for the language task, termed L3E-HD, which enables efficient HDC on low-power edge devices. L3E-HD accelerates the inference by mapping data points to a high-dimensional binary space to simplify similarity search, which dominates costly and frequent operation in HDC. Through marrying HDC with the ensemble technique, L3E-HD also addresses the severe accuracy degradation induced by the compression of the dimension and precision of the model. Our experiments show that the ensemble technique is naturally a perfect fit to boost HDCs. We find that our L3E-HD, which is faster, more efficient, and more accurate than conventional machine learning methods, can even surpass the accuracy of the full-precision model at a smaller model size. Code is released at: https://github.com/MXHX7199/SIGIR22-EnsembleHDC.

Supplementary Material

MP4 File (SIGIR22-sp1222.mp4)
In this paper, we proposed a novel ensemble framework for the language task, termed L3E-HD, which enables efficient HDC on low-power edge devices. L3E-HD accelerates the inference by mapping data points to a high-dimensional binary space to simplify similarity search, which dominates costly and frequent operation in HDC. Through marrying HDC with the ensemble technique, L3E-HD also addresses the severe accuracy degradation induced by the compression of the dimension and precision of the model. Our experiments show that the ensemble technique is naturally a perfect fit to boost HDCs. We find that our L3E-HD, which is faster, more efficient, and more accurate than conventional machine learning methods, can even surpass the accuracy of the full-precision model at a smaller model size.

References

[1]
Túlio C Alberto, Johannes V Lochter, and Tiago A Almeida. 2015. Tubespam: Comment spam filtering on youtube. In 2015 IEEE 14th international conference on machine learning and applications (ICMLA). IEEE, 138--143.
[2]
Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. Nvidia a100 tensor core gpu: Performance and innovation. IEEE Micro 41, 2 (2021), 29--35.
[3]
Loredana Firte, Camelia Lemnaru, and Rodica Potolea. 2010. Spam detection filter using KNN algorithm and resampling. In Proceedings of the 2010 IEEE 6th international conference on intelligent computer communication and processing. IEEE, 27--33.
[4]
Wei Gao and Zhi-Hua Zhou. 2020. Towards convergence rate analysis of random forests for classification. Advances in Neural Information Processing Systems 33 (2020), 9300--9311.
[5]
Trevor Hastie, Saharon Rosset, Ji Zhu, and Hui Zou. 2009. Multi-class adaboost. Statistics and its Interface 2, 3 (2009), 349--360.
[6]
Alejandro Hernandez-Cane, Namiko Matsumoto, Eric Ping, and Mohsen Imani. 2021. Onlinehd: Robust, efficient, and single-pass online learning using hyperdimensional system. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 56--61.
[7]
José María Gómez Hidalgo, Tiago A Almeida, and Akebo Yamakami. 2012. On the validity of a new SMS spam collection. In 2012 11th International Conference on Machine Learning and Applications, Vol. 2. IEEE, 240--245.
[8]
Mohsen Imani, Justin Morris, Samuel Bosch, Helen Shu, Giovanni De Micheli, and Tajana Rosing. 2019. Adapthd: Adaptive efficient training for brain-inspired hyperdimensional computing. In 2019 IEEE Biomedical Circuits and Systems Conference (BioCAS). IEEE, 1--4.
[9]
Mohsen Imani, Justin Morris, John Messerly, Helen Shu, Yaobang Deng, and Tajana Rosing. 2019. Bric: Locality-based encoding for energy-efficient braininspired hyperdimensional computing. In Proceedings of the 56th Annual Design Automation Conference 2019. 1--6.
[10]
Mohsen Imani, Saikishan Pampana, Saransh Gupta, Minxuan Zhou, Yeseong Kim, and Tajana Rosing. 2020. Dual: Acceleration of clustering algorithms using digital-based processing in-memory. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 356--371.
[11]
Mohsen Imani, Zhuowen Zou, Samuel Bosch, Sanjay Anantha Rao, Sahand Salamat, Venkatesh Kumar, Yeseong Kim, and Tajana Rosing. 2021. Revisiting hyperdimensional learning for fpga and low-power architectures. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 221--234.
[12]
Gauri Jain, Manisha Sharma, and Basant Agarwal. 2019. Optimizing semantic LSTM for spam detection. International Journal of Information Technology 11, 2 (2019), 239--250.
[13]
Pentti Kanerva. 2009. Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors. Cognitive computation 1, 2 (2009), 139--159.
[14]
Geethan Karunaratne, Manuel Le Gallo, Giovanni Cherubini, Luca Benini, Abbas Rahimi, and Abu Sebastian. 2020. In-memory hyperdimensional computing. Nature Electronics 3, 6 (2020), 327--337.
[15]
Arman Kazemi, Mohammad Mehdi Sharifi, Zhuowen Zou, Michael Niemier, X Sharon Hu, and Mohsen Imani. 2021. Mimhd: Accurate and efficient hyperdimensional inference using multi-bit in-memory computing. In 2021 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED). IEEE, 1--6.
[16]
Yeseong Kim, Mohsen Imani, Niema Moshiri, and Tajana Rosing. 2020. Geniehd: Efficient dna pattern matching accelerator using hyperdimensional computing. In 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 115--120.
[17]
Denis Kleyko, Abbas Rahimi, Dmitri A Rachkovskij, Evgeny Osipov, and Jan M Rabaey. 2018. Classification and recall with binary hyperdimensional computing: Tradeoffs in choice of density and mapping characteristics. IEEE transactions on neural networks and learning systems 29, 12 (2018), 5880--5898.
[18]
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers. 79--86.
[19]
Fangxin Liu, Wenbo Zhao, Zhezhi He, Yanzhi Wang, Zongwu Wang, Changzhi Dai, Xiaoyao Liang, and Li Jiang. 2021. Improving Neural Network Efficiency via Post-Training Quantization With Adaptive Floating-Point. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5281--5290.
[20]
Sherin Mary Mathews. 2019. Explainable artificial intelligence applications in NLP, biomedical, and malware classification: a literature review. In Intelligent computing-proceedings of the computing conference. Springer, 1269--1292.
[21]
Peer Neubert and Stefan Schubert. 2021. Hyperdimensional computing as a framework for systematic aggregation of image descriptors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16938--16947.
[22]
Nikunj Chandrakant Oza and Stuart Russell. 2001. Online ensemble learning. University of California, Berkeley.
[23]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
[24]
Prathyush Poduval, Zhuowen Zou, Hassan Najafi, Houman Homayoun, and Mohsen Imani. 2021. Stochd: Stochastic hyperdimensional system for efficient and robust learning from raw data. In 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 1195--1200.
[25]
Uwe Quasthoff, Matthias Richter, and Christian Biemann. 2006. Corpus Portal for Search in Monolingual Corpora. In LREC. 1799--1802.
[26]
Abbas Rahimi, Pentti Kanerva, Luca Benini, and Jan M Rabaey. 2018. Efficient biosignal processing using hyperdimensional computing: Network templates for combined learning and classification of ExG signals. Proc. IEEE 107, 1 (2018), 123--143.
[27]
Sahand Salamat, Mohsen Imani, Behnam Khaleghi, and Tajana Rosing. 2019. F5-hd: Fast flexible fpga-based framework for refreshing hyperdimensional computing. In Proceedings of the 2019 ACM/SIGDA International Symposium on Field- Programmable Gate Arrays. 53--62.
[28]
Robert E Schapire. 2003. The boosting approach to machine learning: An overview. Nonlinear estimation and classification (2003), 149--171.
[29]
S Sumathi and Ganesh Kumar Pugalendhi. 2021. Cognition based spam mail text analysis using combined approach of deep neural network classifier and random forest. Journal of Ambient Intelligence and Humanized Computing 12, 6 (2021), 5721--5731.
[30]
Rahul Thapa, Bikal Lamichhane, Dongning Ma, and Xun Jiao. 2021. SpamHD: Memory-Efficient Text Spam Detection using Brain-Inspired Hyperdimensional Computing. In 2021 IEEE Computer Society Annual Symposium on VLSI (ISVLSI). IEEE, 84--89.
[31]
Chi-Yao Tseng and Ming-Syan Chen. 2009. Incremental SVM model for spam detection on dynamic email social networks. In 2009 International Conference on Computational Science and Engineering, Vol. 4. IEEE, 128--135.
[32]
Minxuan Zhou, Muzhou Li, Mohsen Imani, and Tajana Rosing. 2021. Hygraph: Accelerating graph processing with hybrid memory-centric computing. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 330--335.
[33]
Shilin Zhu, Xin Dong, and Hao Su. 2019. Binary ensemble neural network: More bits per network or more networks per bit?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4923--4932.

Cited By

View all
  • (2024)Hyperdimensional Computing vs. Neural Networks: Comparing Architecture and Learning Process2024 25th International Symposium on Quality Electronic Design (ISQED)10.1109/ISQED60706.2024.10528698(1-5)Online publication date: 3-Apr-2024
  • (2024)HyperFeel: An Efficient Federated Learning Framework Using Hyperdimensional ComputingProceedings of the 29th Asia and South Pacific Design Automation Conference10.1109/ASP-DAC58780.2024.10473907(716-721)Online publication date: 22-Jan-2024
  • (2023)Adversarial Attack on Hyperdimensional Computing-based NLP Applications2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE56975.2023.10137289(1-6)Online publication date: Apr-2023
  • Show More Cited By

Index Terms

  1. L3E-HD: A Framework Enabling Efficient Ensemble in High-Dimensional Space for Language Tasks

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval
    July 2022
    3569 pages
    ISBN:9781450387323
    DOI:10.1145/3477495
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 July 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. brain-inspired computing
    2. ensemble learning
    3. hdc

    Qualifiers

    • Short-paper

    Funding Sources

    Conference

    SIGIR '22
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)32
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 28 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Hyperdimensional Computing vs. Neural Networks: Comparing Architecture and Learning Process2024 25th International Symposium on Quality Electronic Design (ISQED)10.1109/ISQED60706.2024.10528698(1-5)Online publication date: 3-Apr-2024
    • (2024)HyperFeel: An Efficient Federated Learning Framework Using Hyperdimensional ComputingProceedings of the 29th Asia and South Pacific Design Automation Conference10.1109/ASP-DAC58780.2024.10473907(716-721)Online publication date: 22-Jan-2024
    • (2023)Adversarial Attack on Hyperdimensional Computing-based NLP Applications2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE56975.2023.10137289(1-6)Online publication date: Apr-2023
    • (2023)Robust Hyperdimensional Computing against Cyber Attacks and Hardware ErrorsProceedings of the 28th Asia and South Pacific Design Automation Conference10.1145/3566097.3568355(598-605)Online publication date: 16-Jan-2023
    • (2023)Hyperdimensional Computing as a Rescue for Efficient Privacy-Preserving Machine Learning-as-a-Service2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD)10.1109/ICCAD57390.2023.10323815(1-8)Online publication date: 28-Oct-2023
    • (2023)HyperNode: An Efficient Node Classification Framework Using HyperDimensional Computing2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD)10.1109/ICCAD57390.2023.10323813(1-9)Online publication date: 28-Oct-2023
    • (2023)HyperGRAF: Hyperdimensional Graph-Based Reasoning Acceleration on FPGA2023 33rd International Conference on Field-Programmable Logic and Applications (FPL)10.1109/FPL60245.2023.00013(34-41)Online publication date: 4-Sep-2023
    • (2023)HyperAttack: An Efficient Attack Framework for HyperDimensional Computing2023 60th ACM/IEEE Design Automation Conference (DAC)10.1109/DAC56929.2023.10247811(1-6)Online publication date: 9-Jul-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media