Skip to main content
SpringerLink
Log in

Search for Sparse Active Inputs: A Review

  • Chapter
Information Theory, Combinatorics, and Search Theory

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 7777))

Abstract

The theory of Compressed Sensing (highly popular in recent years) has a close relative that was developed around thirty years earlier and has been almost forgotten since – the design of screening experiments. For both problems, the main assumption is sparsity of active inputs, and the fundamental feature in both theories is the threshold phenomenon: reliable recovery of sparse active inputs is possible when the rate of design is less than the so-called capacity threshold, and impossible with higher rates.

Another close relative of both theories is multi-access information transmission. We survey a collection of tight and almost tight screening capacity bounds for both adaptive and non-adaptive strategies which correspond to either having or not having feedback in information transmission. These bounds are inspired by results from multi-access capacity theory. We also compare these bounds with the simulated performance of two analysis methods: (i) linear programming relaxation methods akin to basis pursuit used in compressed sensing, and (ii) greedy methods of low complexity for both non-adaptive and adaptive strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahlswede, R.: Multi-way communication channels. In: Proceedings of 2nd International Symposium on Information Theory, Tsahkadzor, pp. 23–52. Akademiai Kiado, Budapest (1971, 1973)

    Google Scholar 

  2. Ahlswede, R., Wegener, I.: Suchprobleme. Teubner, Stuttgart (1979)

    Google Scholar 

  3. Atia, G., Saligrama, V.: Boolean compressed sensing and noisy group testing, arXiv:0907.1061v2 (2009)

    Google Scholar 

  4. Budne, T.A.: Application of random balance designs. Technometrics 1(2), 139–155 (1959)

    Article  MathSciNet  Google Scholar 

  5. Candes, E.J.: Compressive sampling. In: International Congress of Mathematicians, Madrid, Spain, August 22-30, vol. 3, pp. 1433–1452. Eur. Math. Soc., Zürich (2006)

    Google Scholar 

  6. Cover, T., Thomas, J.: Elements of Information Theory. Wiley, N.Y. (1991)

    Book  MATH  Google Scholar 

  7. Csiszár, I., Körner, J.: Information Theory: Coding Theorems for Discrete Memoryless Systems. Academic Press and Akadémiai Kiadó, Budapest (1981)

    MATH  Google Scholar 

  8. Csiszár, I., Shields, P.: Information Theory and Statistics, a Tutorial. In: Foundations and Trend in Communications and Information Theory, vol. 1(4). Now Publishers Inc., Hanover (2004)

    Google Scholar 

  9. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic de-composition by basis pursuit. SIAM J. Scientific Computing 20, 33–61 (1998)

    Article  MATH  Google Scholar 

  10. Donoho, D.L., Elad, M.: Maximal sparsity Representation via L1 Minimization. Proc. Nat. Acad. Sci. 100, 2197–2202 (2003)

    Article  MATH  Google Scholar 

  11. Donoho, D.L., Tanner, J.: Sparse nonnegative solution of underdetermined linear equations by linear programming. Proc. Nat. Acad. Sci. 102(27) (2005)

    Google Scholar 

  12. Dorfman, R.: The detection of defective members of large populations. Ann. Math. Statist. 14(4), 436–440 (1943)

    Article  Google Scholar 

  13. Erlich, Y., Gordon, A., Brand, M., Hannon, G.J., Mitra, P.P.: Compressed genotyping. IEEE Trans. on Inf. Theory 56(2), 706–723 (2010)

    Article  MathSciNet  Google Scholar 

  14. Erdős, P., Rényi, A.: On two problems of information theory. Publ. Math. Inst. of Hung. Acad. of Sci. 8, 229–243 (1963)

    MathSciNet  MATH  Google Scholar 

  15. Gaarder, N., Wolf, J.: The capacity region of a multiple-access discrete memoryless channel can increase with feedback. IEEE Trans. on Inf. Theory 21(1), 100–102 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  16. Gallager, R.G.: Information Theory and Reliable Communication. Wiley, New York (1968)

    MATH  Google Scholar 

  17. Kashin, B.S., Temlyakov, V.N.: A remark on compressed sensing. Mat. Zametki 82(6), 829–937 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  18. Kerkyacharian, G., Mougeot, M., Picard, D., Tribouley, K.: Learning out of leaders. In: Multiscale, Nonlinear and Adaptive Approximation. Springer (2010)

    Google Scholar 

  19. Kolmogorov, A.N.: Three approaches to the quantitative definition of information. Probl. of Inf. Transmiss. 1, 3–11 (1965)

    MATH  Google Scholar 

  20. Leontovich, A.M., Victorova, I.A.: On the number of tests to detect significant variables of a boolean function. Probl. of Inform. Transmiss. 14, 85–97 (1978)

    Google Scholar 

  21. Malioutov, D.M., Cetin, M., Willsky, A.S.: Optimal sparse representations in general overcomplete bases. In: IEEE International Conference on Acoustics, Speech and Signal Processing, Montreal, Canada, vol. 2, pp. 793–796 (May 2004)

    Google Scholar 

  22. Malioutov, D.M., Malyutov, M.B.: Boolean compressed sensing: LP relaxation for group testing. In: IEEE International Conference on Acoustics, Speech and Signal Processing, Kyoto, Japan (March 2012)

    Google Scholar 

  23. Malyutov, M.B.: On planning of screening experiments. In: Proceedings of 1975 IEEE-USSR Workshop on Inf. Theory, pp. 144–147. IEEE Inc., N.Y. (1976)

    Google Scholar 

  24. Malyutov, M.B.: Mathematical models and results in theory of screening experiments. In: Malyutov, M.B. (ed.) Theoretical Problems of Experimental Design, Soviet Radio, Moscow, pp. 5–69 (1977)

    Google Scholar 

  25. Malyutov, M.B.: Separating property of random Matrices. Mat. Zametki 23, 155–167 (1978)

    MathSciNet  MATH  Google Scholar 

  26. Malyutov, M.B.: On the maximal rate of screening designs. Theory Probab. and Appl. 14, 655–657 (1979)

    Google Scholar 

  27. Malyutov, M.B.: On sequential search for significant variables of the additive function. In: Probability Theory and Mathematical Statistics, pp. 298–316. World Scientific, Singapore (1996)

    Google Scholar 

  28. Malyutov, M.B.: Log-efficient search for active variables of linear model. J. of Statist. Plan. and Infer. 100(2), 269–280 (2002)

    Article  MATH  Google Scholar 

  29. Malyutov, M.B.: Compression based homogeneity testing. Doklady of RAS 443(4), 1–4 (2012)

    MATH  Google Scholar 

  30. Malyutov, M.B., D’yachkov, A.G.: On weakly separating designs. In: Methods of Transmission and Processing Information, Nauka, Moscow, pp. 87–104 (1980) (in Russian)

    Google Scholar 

  31. Malyutov, M.B., Mateev, P.S.: Screening designs for non-symmetric response function. Mat. Zametki 27, 109–127 (1980)

    MathSciNet  MATH  Google Scholar 

  32. Malyutov, M.B., Pinsker, M.S.: Note on the simplest model of the random balance method. In: Probabilistic Methods of Research. Moscow University Press (1972) (in Russian)

    Google Scholar 

  33. Malyutov, M.B., Sadaka, H.: Jaynes principle in testing active variables of linear model. Random Operators and Stochastic Equations 6, 311–330 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  34. Malyutov, M.B., Sadaka, H.: Capacity of screening under linear programming analysis. In: Proceedings of the 6th Simulation International Workshop on Simulation, St. Petersburg, Russia, pp. 1041–1045 (2009)

    Google Scholar 

  35. Malyutov, M.B., Tsitovich, I.I.: Sequential search for significant variables of unknown function. Problems of Information Transmission 36(3), 88–107 (1997)

    MATH  Google Scholar 

  36. Malyutov, M.B., Tsitovich, I.I.: Non-parametric search for active inputs of unknown system. In: Multiconference on Systemics, Cybernetics and Informatics, Orlando, FL, July 23-26, vol. 11, pp. 75–83 (2000)

    Google Scholar 

  37. Malyutov, M.B., Wynn, H.P.: Adaptive search for active variables of additive model. In: Markov Processes and Related Fields, pp. 253–265. Birkhauser, Boston (1994)

    Google Scholar 

  38. Meshalkin, L.D.: Justification of the random balance method, pp. 316–318. Industrial Lab. (1970)

    Google Scholar 

  39. Nalimov, V.V., Chernova, N.A.: Statistical Methods of Planning Extremal Experiments, M., Nauka (1965) (in Russian)

    Google Scholar 

  40. Rissanen, J.: Universal coding, information, prediction and estimation. IEEE Trans. Inf. Theory 30(4), 629–636 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  41. Ryabko, B., Astola, J., Malyutov, M.B.: Compression-Based Methods of Prediction and Statistical Analysis of Time Series: Theory and Applications, Tampere, TICSP series, No. 56 (2010)

    Google Scholar 

  42. Srivastava, J.N.: Search for non-negligible variables. Survey of Statistical Design and Linear Models, 507–519 (1975)

    Google Scholar 

  43. Tibshirani, R.: Regression shrinkage and selection via the LASSO. J. Royal. Statist. Soc. B 58, 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  44. Venkataramanan, R., Pradhan, S.S.: A new achievable rate region for the multiple-access channel with noiseless feedback. IEEE Trans. on Inf. Theory 57(12), 8038–8054 (2011)

    Article  MathSciNet  Google Scholar 

  45. Viemeister, J.: Diplomarbeit, Fakulatät für Mathematik, Universität Bielefeld (1982)

    Google Scholar 

  46. Wainwright, M.: Information-theoretic bounds on sparsity recovery in the high-dimensional and noisy setting. In: Proc. 2007 IEEE International Symposium on Information Theory, Piscataway, pp. 961–965. IEEE Press, NJ (2007)

    Chapter  Google Scholar 

  47. Ziv, J.: On classification and universal data compression. IEEE Trans. on Inf. Theory 34(2), 278–286 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  48. Zubashich, V.F., Lysyansky, A.V., Malyutov, M.B.: Block–randomized distributed trouble–shooting construction in large circuits with redundancy. Izvestia of the USSR Acad. of Sci., Technical Cybernetics (6) (1976)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Math. Dept., Northeastern University, 360 Huntington Ave., Boston, MA, 02115, USA

    Mikhail Malyutov

Authors
  1. Mikhail Malyutov
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Mathematics, University of Bielefeld, Universitätsstr. 25, 33615, Bielefeld, Germany

    Harout Aydinian  & Christian Deppe  & 

  2. Department of Computer Science, University of Salerno, via Ponte don Melillo, 84084, Fisciano, Italy

    Ferdinando Cicalese

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Malyutov, M. (2013). Search for Sparse Active Inputs: A Review. In: Aydinian, H., Cicalese, F., Deppe, C. (eds) Information Theory, Combinatorics, and Search Theory. Lecture Notes in Computer Science, vol 7777. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36899-8_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-36899-8_31

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-36898-1

  • Online ISBN: 978-3-642-36899-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation