Abstract
In the information theoretic world, entropy is both the measure of randomness in a source and a bound for the compression achievable for that source by any encoding scheme. But when we must restrict ourselves to efficient schemes, entropy no longer captures these notions well. For example, there are distributions with very low entropy that nonetheless look random for polynomial-bound algorithms.
Different notions of computational entropy have been proposed to take the role of entropy in such settings. Results in Goldberg and Sipser (SIAM J. Comput. 20(3):524–536, 1991) and Wee (IEEE conference on computational complexity, pp. 29–41, 2004) suggest that when time bounds are introduced, the entropy of a distribution no longer coincides with the most effective compression for that source.
This paper analyses three measures that try to capture the compressibility of a source, establishing relations and separations between them and analysing the two special cases of the uniform and the universal distribution m t over binary strings of a fixed size. It is shown that for the uniform distribution the three measures are equivalent and that for m t there is a clear separation between metric type entropy and the maximum compressibility of a source.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Barak, B., Shaltiel, R., Widgerson, A.: Computational analogues of entropy. In: Proceedings of the 7th Conference on Randomization and Computation, (RANDOM) (2003). Available at http://www.math.ias.edu/~avi/PUBLICATIONS/MYPAPERS/BSW03/bsw03.ps
Corless, R.M., Gonnet, G.H., Hare, D.E.G., Jeffrey, D.J., Knuth, D.E.: On the Lambert W function. Adv. Comput. Math. 5, 329–359 (1996)
Goldberg, A., Sipser, M.: Compression and ranking. SIAM J. Comput. 20(3), 524–536 (1991)
Grünwald, P., Vitányi, P.: Kolmogorov complexity and information theory. J. Logic, Lang. Inf. 12(4), 497–529 (2003). Available at http://citeseer.ist.psu.edu/565384.html
Håstad, J., Impagliazzo, R., Levin, L., Luby, M.: A pseudorandom generator from any one-way function. SIAM J. Comput. 28(4), 1364–1396 (1999). Available at http://citeseer.ist.psu.edu/hastad99pseudorandom.html
Hsiao, C.-Y., Lu, C.-J., Reyzin, L.: Conditional computational entropy, or toward separating pseudoentropy from compressibility. In: Eurocrypt 2007, Proceedings, pp. 169–186 (2007)
Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and Its Applications, 2nd edn. Springer, Berlin (1997)
Nisan, N., Wigderson, A.: Hardness vs. randomness. J. Comput. Syst. Sci 49(2), 149–167 (1994)
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423, 623–656 (1948)
Trevisan, L., Vadhan, S., Zuckerman, D.: Compression of samplable sources. Comput. Complex. 14(3), 186–227 (2005)
Wee, H.: On pseudoentropy versus compressibility. In: IEEE Conference On Computational Complexity, pp. 29–41 (2004)
Yao, A.: Theory and applications of trapdoor functions (Extended abstract). In: Foundations of Computer Science, pp. 80–91 (1982)
Author information
Authors and Affiliations
Corresponding author
Additional information
Partially supported by KCrypt (POSC/EIA/60819/2004), the grant SFRH/BD/13124/2003 from FCT and funds granted to LIACC through the Programa de Financiamento Plurianual, FCT and Programa POSI.
Rights and permissions
About this article
Cite this article
Pinto, A. Comparing Notions of Computational Entropy. Theory Comput Syst 45, 944–962 (2009). https://doi.org/10.1007/s00224-009-9177-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00224-009-9177-7