PLANC: Parallel Low-rank Approximation with Nonnegativity Constraints

Eswar, Srinivas; Hayashi, Koby; Ballard, Grey; Kannan, Ramakrishnan; Matheson, Michael A.; Park, Haesun

doi:10.1145/3432185

Title: PLANC: Parallel Low-rank Approximation with Nonnegativity Constraints

Journal Article · Fri Jun 25 00:00:00 EDT 2021 · ACM Transactions on Mathematical Software

DOI:https://doi.org/10.1145/3432185· OSTI ID:1820808

Eswar, Srinivas ^[1]; Hayashi, Koby ^[1]; Ballard, Grey ^[2];

^[3];

^[3]; Park, Haesun ^[1]

Georgia Institute of Technology, Atlanta, GA (United States)
Wake Forest Univ., Winston-Salem, NC (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

In this work, we consider the problem of low-rank approximation of massive dense nonnegative tensor data, for example, to discover latent patterns in video and imaging applications. As the size of data sets grows, single workstations are hitting bottlenecks in both computation time and available memory. We propose a distributed-memory parallel computing solution to handle massive data sets, loading the input data across the memories of multiple nodes, and performing efficient and scalable parallel algorithms to compute the low-rank approximation. We present a software package called Parallel Low-rank Approximation with Nonnegativity Constraints, which implements our solution and allows for extension in terms of data (dense or sparse, matrices or tensors of any order), algorithm (e.g., from multiplicative updating techniques to alternating direction method of multipliers), and architecture (we exploit GPUs to accelerate the computation in this work). We describe our parallel distributions and algorithms, which are careful to avoid unnecessary communication and computation, show how to extend the software to include new algorithms and/or constraints, and report efficiency and scalability results for both synthetic and real-world data sets.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC); National Science Foundation (NSF)

Grant/Contract Number:: AC05-00OR22725; OAC-1642385; OAC-1642410; AC02-05CH11231; SC0020347

OSTI ID:: 1820808

Journal Information:: ACM Transactions on Mathematical Software, Vol. 47, Issue 3; ISSN 0098-3500

Publisher:: Association for Computing MachineryCopyright Statement

Country of Publication:: United States

Language:: English

References (39)

Randomized nonnegative matrix factorization Erichson, N. Benjamin; Mendible, Ariana; Wihlborn, Sophie Pattern Recognition Letters, Vol. 104 https://doi.org/10.1016/j.patrec.2018.01.007	journal	March 2018
Fast Nonnegative Matrix Factorization: An Active-Set-Like Method and Comparisons Kim, Jingu; Park, Haesun SIAM Journal on Scientific Computing, Vol. 33, Issue 6 https://doi.org/10.1137/110821172	journal	January 2011
Numerical tensor calculus Hackbusch, Wolfgang Acta Numerica, Vol. 23 https://doi.org/10.1017/S0962492914000087	journal	May 2014
Interior-Point Gradient Method for Large-Scale Totally Nonnegative Least Squares Problems Merritt, M.; Zhang, Y. Journal of Optimization Theory and Applications, Vol. 126, Issue 1 https://doi.org/10.1007/s10957-005-2668-z	journal	July 2005
Nonnegative Matrix and Tensor Factorization [Lecture Notes] Cichocki, A.; Zdunek, R.; Amari, S. -i. IEEE Signal Processing Magazine, Vol. 25, Issue 1 https://doi.org/10.1109/MSP.2008.4408452	journal	January 2008
A fast non-negativity-constrained least squares algorithm Bro, Rasmus; De Jong, Sijmen Journal of Chemometrics, Vol. 11, Issue 5 https://doi.org/10.1002/(SICI)1099-128X(199709/10)11:5<393::AID-CEM483>3.0.CO;2-L	journal	September 1997
A comparison of algorithms for fitting the PARAFAC model Tomasi, Giorgio; Bro, Rasmus Computational Statistics & Data Analysis, Vol. 50, Issue 7 https://doi.org/10.1016/j.csda.2004.11.013	journal	April 2006
Collective communication: theory, practice, and experience Chan, Ernie; Heimlich, Marcel; Purkayastha, Avi Concurrency and Computation: Practice and Experience, Vol. 19, Issue 13 https://doi.org/10.1002/cpe.1206	journal	January 2007
A Flexible and Efficient Algorithmic Framework for Constrained Matrix and Tensor Factorization Huang, Kejun; Sidiropoulos, Nicholas D.; Liavas, Athanasios P. IEEE Transactions on Signal Processing, Vol. 64, Issue 19 https://doi.org/10.1109/TSP.2016.2576427	journal	October 2016
MPI-FAUN: An MPI-Based Framework for Alternating-Updating Nonnegative Matrix Factorization Kannan, Ramakrishnan; Ballard, Grey; Park, Haesun IEEE Transactions on Knowledge and Data Engineering, Vol. 30, Issue 3 https://doi.org/10.1109/TKDE.2017.2767592	journal	March 2018
Projected Gradient Methods for Nonnegative Matrix Factorization Lin, Chih-Jen Neural Computation, Vol. 19, Issue 10 https://doi.org/10.1162/neco.2007.19.10.2756	journal	October 2007
Hierarchical Singular Value Decomposition of Tensors Grasedyck, Lars SIAM Journal on Matrix Analysis and Applications, Vol. 31, Issue 4 https://doi.org/10.1137/090764189	journal	January 2010
A weighted non-negative least squares algorithm for three-way ‘PARAFAC’ factor analysis Paatero, Pentti Chemometrics and Intelligent Laboratory Systems, Vol. 38, Issue 2 https://doi.org/10.1016/S0169-7439(97)00031-2	journal	October 1997
Tensor Decomposition for Signal Processing and Machine Learning Sidiropoulos, Nicholas D.; De Lathauwer, Lieven; Fu, Xiao IEEE Transactions on Signal Processing, Vol. 65, Issue 13 https://doi.org/10.1109/TSP.2017.2690524	journal	July 2017
Exploiting Efficient Representations in Large-Scale Tensor Decompositions Vervliet, Nico; Debals, Otto; De Lathauwer, Lieven SIAM Journal on Scientific Computing, Vol. 41, Issue 2 https://doi.org/10.1137/17M1152371	journal	January 2019
Tensor Decompositions and Applications Kolda, Tamara G.; Bader, Brett W. SIAM Review, Vol. 51, Issue 3 https://doi.org/10.1137/07070111X	journal	August 2009
Computing Dense Tensor Decompositions with Optimal Dimension Trees Kaya, Oguz; Robert, Yves Algorithmica, Vol. 81, Issue 5 https://doi.org/10.1007/s00453-018-0525-3	journal	October 2018
Long-Term Optical Access to an Estimated One Million Neurons in the Live Mouse Cortex Kim, Tony Hyun; Zhang, Yanping; Lecoq, Jérôme Cell Reports, Vol. 17, Issue 12 https://doi.org/10.1016/j.celrep.2016.12.004	journal	December 2016
Positive tensor factorization Welling, Max; Weber, Markus Pattern Recognition Letters, Vol. 22, Issue 12 https://doi.org/10.1016/S0167-8655(01)00070-8	journal	October 2001
Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers Boyd, Stephen; Parikh, Neal; Chu, Eric https://doi.org/10.1561/2200000016	book	January 2011
Faster least squares approximation Drineas, Petros; Mahoney, Michael W.; Muthukrishnan, S. Numerische Mathematik, Vol. 117, Issue 2 https://doi.org/10.1007/s00211-010-0331-6	journal	October 2010
Fast algorithm for the solution of large-scale non-negativity-constrained least squares problems Van Benthem, Mark H.; Keenan, Michael R. Journal of Chemometrics, Vol. 18, Issue 10 https://doi.org/10.1002/cem.889	journal	October 2004
A massively parallel tensor contraction framework for coupled-cluster computations Solomonik, Edgar; Matthews, Devin; Hammond, Jeff R. Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.06.002	journal	December 2014
Optimization of Collective Communication Operations in MPICH Thakur, Rajeev; Rabenseifner, Rolf; Gropp, William The International Journal of High Performance Computing Applications, Vol. 19, Issue 1 https://doi.org/10.1177/1094342005051521	journal	February 2005
PARAFAC algorithms for large-scale problems Huy Phan, Anh; Cichocki, Andrzej Neurocomputing, Vol. 74, Issue 11 https://doi.org/10.1016/j.neucom.2010.06.030	journal	May 2011
Nesterov-Based Alternating Optimization for Nonnegative Tensor Factorization: Algorithm and Parallel Implementation Liavas, Athanasios P.; Kostoulas, Georgios; Lourakis, Georgios IEEE Transactions on Signal Processing, Vol. 66, Issue 4 https://doi.org/10.1109/TSP.2017.2777399	journal	February 2018
A Practical Randomized CP Tensor Decomposition Battaglino, Casey; Ballard, Grey; Kolda, Tamara G. SIAM Journal on Matrix Analysis and Applications, Vol. 39, Issue 2 https://doi.org/10.1137/17M1112303	journal	January 2018
Efficient MATLAB Computations with Sparse and Factored Tensors Bader, Brett W.; Kolda, Tamara G. SIAM Journal on Scientific Computing, Vol. 30, Issue 1 https://doi.org/10.1137/060676489	journal	January 2008
Parallel Nonnegative CP Decomposition of Dense Tensors Ballard, Grey; Hayashi, Koby; Ramakrishnan, Kannan 2018 IEEE 25th International Conference on High Performance Computing (HiPC) https://doi.org/10.1109/HiPC.2018.00012	conference	December 2018
Communication Lower Bounds for Matricized Tensor Times Khatri-Rao Product Ballard, Grey; Knight, Nicholas; Rouse, Kathryn 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2018.00065	conference	May 2018
Efficient and scalable computations with sparse tensors Baskaran, Muthu; Meister, Benoit; Vasilache, Nicolas 2012 IEEE Conference on High Performance Extreme Computing (HPEC) https://doi.org/10.1109/HPEC.2012.6408676	conference	September 2012
Distributed non-negative matrix factorization with determination of the number of latent features Chennupati, Gopinath; Vangara, Raviteja; Skau, Erik The Journal of Supercomputing https://doi.org/10.1007/s11227-020-03181-6	journal	February 2020
Block-randomized Stochastic Proximal Gradient for Constrained Low-rank Tensor Factorization Fu, Xiao; Gao, Cheng; Wai, Hoi-To ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP.2019.8682465	conference	May 2019
A high-performance parallel algorithm for nonnegative matrix factorization Kannan, Ramakrishnan; Ballard, Grey; Park, Haesun ACM SIGPLAN Notices, Vol. 51, Issue 8 https://doi.org/10.1145/3016078.2851152	journal	February 2016
High Performance Parallel Algorithms for the Tucker Decomposition of Sparse Tensors Kaya, Oguz; Ucar, Bora 2016 45th International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2016.19	conference	August 2016
Parallel Candecomp/Parafac Decomposition of Sparse Tensors Using Dimension Trees Kaya, Oguz; Uçar, Bora SIAM Journal on Scientific Computing, Vol. 40, Issue 1 https://doi.org/10.1137/16M1102744	journal	January 2018
Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework Kim, Jingu; He, Yunlong; Park, Haesun Journal of Global Optimization, Vol. 58, Issue 2 https://doi.org/10.1007/s10898-013-0035-4	journal	March 2013
Fast Alternating LS Algorithms for High Order CANDECOMP/PARAFAC Tensor Factorizations Phan, Anh-Huy; Tichavsky, Petr; Cichocki, Andrzej IEEE Transactions on Signal Processing, Vol. 61, Issue 19 https://doi.org/10.1109/TSP.2013.2269903	journal	October 2013
SPLATT: Efficient and Parallel Sparse Tensor-Matrix Multiplication Smith, Shaden; Ravindran, Niranjay; Sidiropoulos, Nicholas D. 2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2015.27	conference	May 2015

Similar Records

MPI-FAUN: An MPI-Based Framework for Alternating-Updating Nonnegative Matrix Factorization

Journal Article · Mon Oct 30 00:00:00 EDT 2017 · IEEE Transactions on Knowledge and Data Engineering · OSTI ID:1820808

Kannan, Ramakrishnan; Ballard, Grey; Park, Haesun

Parallel Nonnegative CP Decomposition of Dense Tensors

Conference · Sat Dec 01 00:00:00 EST 2018 · OSTI ID:1820808

Ballard, Grey; Hayashi, Koby; Kannan, Ramakrishnan {ramki}

TuckerMPI: A Parallel C++/MPI Software Package for Large-scale Data Compression via the Tucker Tensor Decomposition

Journal Article · Thu Jun 11 00:00:00 EDT 2020 · ACM Transactions on Mathematical Software · OSTI ID:1820808

Ballard, Grey; Klinvex, Alicia; Kolda, Tamara G.

Related Subjects

97 MATHEMATICS AND COMPUTING

Title: PLANC: Parallel Low-rank Approximation with Nonnegativity Constraints

Citation Formats

References (39)

Similar Records

Related Subjects