skip to main content
10.1145/3313276.3316363acmconferencesArticle/Chapter ViewAbstractPublication PagesstocConference Proceedingsconference-collections
research-article

A universal sampling method for reconstructing signals with simple Fourier transforms

Published: 23 June 2019 Publication History

Abstract

Reconstructing continuous signals based on a small number of discrete samples is a fundamental problem across science and engineering. We are often interested in signals with "simple'' Fourier structure -- e.g., those involving frequencies within a bounded range, a small number of frequencies, or a few blocks of frequencies -- i.e., bandlimited, sparse, and multiband signals, respectively. More broadly, any prior knowledge on a signal's Fourier power spectrum can constrain its complexity. Intuitively, signals with more highly constrained Fourier structure require fewer samples to reconstruct.
We formalize this intuition by showing that, roughly, a continuous signal from a given class can be approximately reconstructed using a number of samples proportional to the statistical dimension of the allowed power spectrum of that class. We prove that, in nearly all settings, this natural measure tightly characterizes the sample complexity of signal reconstruction.
Surprisingly, we also show that, up to log factors, a universal non-uniform sampling strategy can achieve this optimal complexity for any class of signals. We present an efficient and general algorithm for recovering a signal from the samples taken. For bandlimited and sparse signals, our method matches the state-of-the-art, while providing the the first computationally and sample efficient solution to a broader range of problems, including multiband signal reconstruction and Gaussian process regression tasks in one dimension.
Our work is based on a novel connection between randomized linear algebra and the problem of reconstructing signals with constrained Fourier structure. We extend tools based on statistical leverage score sampling and column-based matrix reconstruction to the approximation of continuous linear operators that arise in the signal reconstruction problem. We believe these extensions are of independent interest and serve as a foundation for tackling a broad range of continuous time problems using randomized methods.

References

[1]
Ahmed Alaoui and Michael W. Mahoney. 2015. Fast Randomized Kernel Ridge Regression with Statistical Guarantees. In NIPS 2015. 775–783.
[2]
Haim Avron, Kenneth L. Clarkson, and David P. Woodruff. 2017. Sharper Bounds for Regularized Data Fitting. In RANDOM 2017.
[3]
Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. 2017. Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guarantees. In ICML 2017.
[4]
Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandieh. 2018. A Universal Sampling Method for Reconstructing Signals with Simple Fourier Transforms. (2018). arXiv: 1812.08723
[5]
Francis Bach. 2017. On the Equivalence Between Kernel Quadrature Rules and Random Feature Expansions. Journal of Machine Learning Research 18, 21 (2017).
[6]
Joshua Batson, Daniel Spielman, and Nikhil Srivastava. 2014. Twice-Ramanujan Sparsifiers. SIAM Rev. 56, 2 (2014), 315–334.
[7]
Peter Borwein and Tamás Erdélyi. 2012.
[8]
Polynomials and Polynomial Inequalities. Vol. 161. Springer Science & Business Media.
[9]
Marc Bourgeois, Frank T. A. W. Wajer, Dirk van Ormondt, and Danielle Graveron-Demilly. 2001.
[10]
Reconstruction of MRI Images from Non-Uniform Sampling and Its Application to Intrascan Motion Correction in Functional MRI. Birkhäuser Boston.
[11]
Christos Boutsidis, Michael W. Mahoney, and Petros Drineas. 2009. An Improved approximation algorithm for the column subset selection problem. In SODA 2009.
[12]
Yoram Bresler and Alber Macovski. 1986. Exact Maximum Likelihood Parameter Estimation of Superimposed Exponential Signals in Noise. IEEE Transactions on Acoustics, Speech, and Signal Processing 34, 5 (1986), 1081–1089.
[13]
Daniele Calandriello, Alessandro Lazaric, and Michal Valko. 2016. Analysis of Nyström Method with Sequential Ridge Leverage Score Sampling. In UAI 2016.
[14]
Xue Chen, Daniel M. Kane, Eric Price, and Zhao Song. 2016.
[15]
Fourier-Sparse Interpolation without a Frequency Gap. In FOCS 2016. 741–750.
[16]
Xue Chen and Eric Price. 2018. Active Regression via Linear-Sample Sparsification. arXiv 1711.10051 (2018).
[17]
Kenneth L. Clarkson and David P. Woodruff. 2013. Low Rank Approximation and Regression in Input Sparsity Time. In STOC 2013. 81–90.
[18]
Albert Cohen, Mark A. Davenport, and Dany Leviatan. 2013. On the Stability and Accuracy of Least Squares Approximations. Foundations of Computational Mathematics 13, 5 (01 Oct 2013), 819–834.
[19]
Michael B. Cohen, Cameron Musco, and Christopher Musco. 2017. Input Sparsity Time Low-Rank Approximation via Ridge Leverage Score Sampling. In SODA 2017.
[20]
Michael B. Cohen, Jelani Nelson, and David P. Woodruff. 2016. Optimal Approximate Matrix Product in Terms of Stable Rank. In 43rd International Colloquium on Automata, Languages, and Programming (ICALP 2016). Dagstuhl, Germany.
[21]
Gaspard Riche de Prony. 1795. Essay experimental et analytique: sur les lois de la dilatabilite de fluides elastique et sur celles de la force expansive de la vapeur de l’alcool, a differentes temperatures. Journal de l’Ecole Polytechnique (1795).
[22]
Amit Deshpande and Luis Rademacher. 2010.
[23]
Efficient Volume Sampling for Row/Column Subset Selection. In FOCS 2010. 329–338.
[24]
David L. Donoho. 2006. Compressed sensing. IEEE Transactions on Information Theory 52, 4 (2006), 1289–1306.
[25]
Petros Drineas and Michael W. Mahoney. 2016. RandNLA: Randomized Numerical Linear Algebra. Commun. ACM 59, 6 (2016).
[26]
Yonina C. Eldar. 2015.
[27]
Sampling Theory: Beyond Bandlimited Systems (1st ed.). Cambridge University Press, New York, NY, USA.
[28]
Yonina C. Eldar and Michael Unser. 2006. Nonideal Sampling and Interpolation from Noisy Observations in Shift-invariant Spaces. IEEE Transactions on Signal Processing 54, 7 (2006), 2636–2651.
[29]
Karl J. Friston, Peter Jezzard, and Robert Turner. 1994. Analysis of Functional MRI Time-series. Human Brain Mapping 1, 2 (1994), 153–171.
[30]
Miha Fuderer. 1989. Ringing Artifact Reduction by an Efficient Likelihood Improvement Method. In Science and Engineering of Medical Imaging, Vol. 1137.
[31]
Mark S. Handcock and Michael L. Stein. 1993. A Bayesian Analysis of Kriging. Technometrics 35, 4 (1993), 403–410.
[32]
Godfrey Harold Hardy and John Edensor Littlewood. 1914. Some Problems of Diophantine Approximation. Acta mathematica 37, 1 (1914), 155–191.
[33]
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2002.
[34]
The Elements of Statistical Learning: Data Mining, Inference and Prediction (2nd ed.). Springer.
[35]
John K. Hunter and Bruno Nachtergaele. 2001.
[36]
Applied Analysis.
[37]
Environmental Systems Research Institute. 2018. ArcGIS Desktop: Release 10.
[38]
Santhosh Karnik, Zhihui Zhu, Michael B. Wakin, Justin Romberg, and Mark A. Davenport. 2017.
[39]
The Fast Slepian Transform. Applied and Computational Harmonic Analysis (2017).
[40]
Vladimir A. Kotelnikov. 1933. On the Carrying Capacity of the Ether and Wire in Telecommunications. Material for the First All-Union Conference on Questions of Communication, Izd. Red. Upr. Svyazi RKKA (1933).
[41]
Henry J. Landau. 1967. Sampling, Data Transmission, and the Nyquist Rate. Proc. IEEE 55, 10 (1967), 1701–1706.
[42]
Henry J. Landau and Henry O. Pollak. 1961. Prolate Spheroidal Wave Functions, Fourier Analysis and Uncertainty – II. The Bell System Technical Journal (1961).
[43]
Henry J. Landau and Henry O. Pollak. 1962. Prolate Spheroidal Wave Functions, Fourier Analysis and Uncertainty – III: The Dimension of the Space of Essentially Time- and Band-limited Signals. The Bell System Technical Journal 41, 4 (1962).
[44]
Alan H. Lettington and Qi He Hong. 1995. Image Restoration Using a Lorentzian Probability Model. Journal of Modern Optics 42, 7 (1995), 1367–1376.
[45]
Moshe Mishali and Yonina C. Eldar. 2009. Blind Multiband Signal Reconstruction: Compressed Sensing for Analog Signals. IEEE Trans. on Signal Processing (2009).
[46]
Moshe Mishali and Yonina C. Eldar. 2010. From Theory to Practice: Sub-Nyquist Sampling of Sparse Wideband Analog Signals. IEEE Journal of Selected Topics in Signal Processing 4 (2010), 375–391.
[47]
Ankur Moitra. 2015. Super-resolution, Extremal Functions and the Condition Number of Vandermonde Matrices. In STOC 2015. 821–830.
[48]
Cameron Musco and Christopher Musco. 2017.
[49]
Recursive Sampling for the Nyström Method. In NIPS 2017. 3833–3845.
[50]
Cameron Musco and David P. Woodruff. 2017. Sublinear Time Low-Rank Approximation of Positive Semidefinite Matrices. FOCS 2017 (2017).
[51]
Paul Nevai. 1986. Géza Freud, Orthogonal Polynomials and Christoffel Functions. A case study. Journal of Approximation Theory 48, 1 (1986), 3–167.
[52]
Harry Nyquist. 1928. Certain Topics in Telegraph Transmission Theory. Transactions of the American Institute of Electrical Engineers 47, 2 (1928), 617–644.
[53]
Edouard Pauwels, Francis Bach, and Jean-Philippe Vert. 2018. Relating Leverage Scores and Density using Regularized Christoffel Functions. In NIPS 2018.
[54]
Béatrices Pesquet-Popescu and Jacques L. Vehel. 2002. Stochastic fractal models for image processing. IEEE Signal Processing Magazine 19, 5 (2002), 48–62.
[55]
Vladilen F. Pisarenko. 1973.
[56]
The Retrieval of Harmonics from a Covariance Function. Geophysical Journal International 33, 3 (1973), 347–366.
[57]
Eric Price and Zhao Song. 2015.
[58]
A Robust Sparse Fourier Transform in the Continuous Setting. In FOCS 2015. 583–600.
[59]
Sathish Ramani, Dimitri van de Ville, and Michael Unser. 2005.
[60]
Sampling in Practice: is the Best Reconstruction Space Bandlimited?. In IEEE International Conference on Image Processing.
[61]
S Ramani, D Van De Ville, and M Unser. 2006. Non-Ideal Sampling and Adapted Reconstruction Using the Stochastic Matern Model. In ICASSP 2006.
[62]
Carl Edward Rasmussen and Christopher K. I. Williams. 2006.
[63]
Gaussian Processes for Machine Learning. The MIT Press.
[64]
B. Ripley. 1989.
[65]
Statistical Inference for Spatial Processes. Cambridge Univ. Press.
[66]
B. Ripley. 2005.
[67]
Spatial statistics. John Wiley & Sons.
[68]
Tamas Sarlos. 2006. Improved Approximation Algorithms for Large Matrices via Random Projections. In FOCS 2006. 143–152.
[69]
Claude E. Shannon. 1949. Communication in the Presence of Noise. Proceedings of the Institute of Radio Engineers 37, 1 (1949), 10–21.
[70]
John Shawe-Taylor and Nello Cristianini. 2004.
[71]
Kernel Methods for Pattern Analysis. Cambridge University Press.
[72]
Yoel Shkolnisky, Mark Tygert, and Vladimir Rokhlin. 2006. Approximation of Bandlimited Functions. Applied and Computational Harmonic Analysis (2006).
[73]
David Slepian and Henry O. Pollak. 1961.
[74]
Prolate spheroidal wave functions, Fourier analysis and uncertainty – I. The Bell System Technical Journal (1961).
[75]
A Universal Sampling Method for Reconstructing Signals with Simple Fourier Transforms STOC ’19, June 23–26, 2019, Phoenix, AZ, USA
[76]
Daniel A. Spielman and Nikhil Srivastava. 2011. Graph Sparsification by Effective Resistances. SIAM J. Comput. 40, 6 (2011), 1913–1926. STOC 2008.
[77]
Vilmos Totik. 2000. Asymptotics for Christoffel Functions for General Measures on the Real Line. Journal d’Analyse Mathématique 81, 1 (2000), 283–303.
[78]
E T. Whittaker. 1915. On the Functions Which are Represented by the Expansions of the Interpolation Theory. Proc. of the Royal Soc. of Edinburgh (1915).
[79]
Virginia Vassilevska Williams. 2012.
[80]
Multiplying Matrices Faster than Coppersmith-Winograd. In STOC 2012. ACM, 887–898.
[81]
Keith J. Worsley, Sean Marrett, Peter Neelin, Alain C. Vandal, Karl J. Friston, and Alan C. Evans. 1996. A Unified Statistical Approach for Determining Significant Signals in Images of Cerebral Activation. Human Brain Mapping 4, 1 (1996).
[82]
Hong Xiao. 2001.
[83]
Prolate Spheroidal Wavefunctions, Quadrature, Interpolation, and Asymptotic Formulae. Ph.D. Dissertation. Yale University.
[84]
Hong Xiao, Vladimir Rokhlin, and Norman Yarvin. 2001.
[85]
Prolate Spheroidal Wavefunctions, Quadrature and Interpolation. Inverse Problems 17, 4 (2001).
[86]
Tong Zhang. 2005. Learning bounds for kernel regression using effective data dimensionality. Neural Computation 17, 9 (2005), 2077–2098.

Cited By

View all
  • (2024)Teachers’ Commitment and Their Performance in Naawan District, Misamis OrientalEuropean Modern Studies Journal10.59573/emsj.8(3).2024.198:3(259-289)Online publication date: 12-Jul-2024
  • (2024)A unified framework for learning with nonlinear model classes from arbitrary linear samplesProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692077(169-202)Online publication date: 21-Jul-2024
  • (2024)Advancements in Remote Compressive Hyperspectral Imaging: Adaptive Sampling with Low-Rank Tensor Image ReconstructionElectronics10.3390/electronics1314269813:14(2698)Online publication date: 10-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
STOC 2019: Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing
June 2019
1258 pages
ISBN:9781450367059
DOI:10.1145/3313276
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 June 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Leverage score sampling
  2. Numerical linear algebra
  3. Signal reconstruction

Qualifiers

  • Research-article

Conference

STOC '19
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,469 of 4,586 submissions, 32%

Upcoming Conference

STOC '25
57th Annual ACM Symposium on Theory of Computing (STOC 2025)
June 23 - 27, 2025
Prague , Czech Republic

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)44
  • Downloads (Last 6 weeks)11
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Teachers’ Commitment and Their Performance in Naawan District, Misamis OrientalEuropean Modern Studies Journal10.59573/emsj.8(3).2024.198:3(259-289)Online publication date: 12-Jul-2024
  • (2024)A unified framework for learning with nonlinear model classes from arbitrary linear samplesProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692077(169-202)Online publication date: 21-Jul-2024
  • (2024)Advancements in Remote Compressive Hyperspectral Imaging: Adaptive Sampling with Low-Rank Tensor Image ReconstructionElectronics10.3390/electronics1314269813:14(2698)Online publication date: 10-Jul-2024
  • (2023)mmEcho: A mmWave-based Acoustic Eavesdropping Method2023 IEEE Symposium on Security and Privacy (SP)10.1109/SP46215.2023.10179484(1840-1856)Online publication date: May-2023
  • (2022)Robust model selection and nearly-proper learning for GMMsProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3601929(22830-22843)Online publication date: 28-Nov-2022
  • (2022) Active Linear Regression for ℓ p Norms and Beyond 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS)10.1109/FOCS54457.2022.00076(744-753)Online publication date: Oct-2022
  • (2020)The statistical cost of robust kernel hyperparameter tuningProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3497515(21334-21345)Online publication date: 6-Dec-2020
  • (2020)Generalized leverage score sampling for neural networksProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3496628(10775-10787)Online publication date: 6-Dec-2020
  • (2020)Fourier sparse leverage scores and approximate kernel learningProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3495734(109-122)Online publication date: 6-Dec-2020
  • (2020)Randomized numerical linear algebra: Foundations and algorithmsActa Numerica10.1017/S096249292000002129(403-572)Online publication date: 30-Nov-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media