The Function-Inversion Problem: Barriers and Opportunities

Abstract

The task of function inversion is central to cryptanalysis: breaking block ciphers, forging signatures, and cracking password hashes are all special cases of the function-inversion problem. In 1980, Hellman showed that it is possible to invert a random function $f{:}\,[N] \rightarrow [N]$ in time $T = \widetilde{O}(N^{2/3})$ given only $S = \widetilde{O}(N^{2/3})$ bits of precomputed advice about f. Hellman’s algorithm is the basis for the popular “Rainbow Tables” technique (Oechslin 2003), which achieves the same asymptotic cost and is widely used in practical cryptanalysis.

Is Hellman’s method the best possible algorithm for inverting functions with preprocessed advice? The best known lower bound, due to Yao (1990), shows that $ST = \widetilde{\Omega }(N)$, which still admits the possibility of an $S = T = \widetilde{O}(N^{1/2})$ attack. There remains a long-standing and vexing gap between Hellman’s $N^{2/3}$ upper bound and Yao’s $N^{1/2}$ lower bound. Understanding the feasibility of an $S = T = N^{1/2}$ algorithm is cryptanalytically relevant since such an algorithm could perform a key-recovery attack on AES-128 in time $2^{64}$ using a precomputed table of size $2^{64}$.

For the past 29 years, there has been no progress either in improving Hellman’s algorithm or in strengthening Yao’s lower bound. In this work, we connect function inversion to problems in other areas of theory to (1) explain why progress may be difficult and (2) explore possible ways forward.

Our results are as follows:

• We show that any improvement on Yao’s lower bound on function-inversion algorithms will imply new lower bounds on depth-two circuits with arbitrary gates. Further, we show that proving strong lower bounds on non-adaptive function-inversion algorithms would imply breakthrough circuit lower bounds on linear-size log-depth circuits.

• We take first steps towards the study of the injective function-inversion problem, which has manifold cryptographic applications. In particular, we show that improved algorithms for breaking PRGs with preprocessing would give improved algorithms for inverting injective functions with preprocessing.

• Finally, we show that function inversion is closely related to well-studied problems in communication complexity and data structures. Through these connections we immediately obtain the best known algorithms for problems in these domains.

1 Introduction

A central task in cryptanalysis is that of function inversion. That is, given a function $f{:}\,[N] \rightarrow [N]$ and a point $y \in [N]$, find a value $x \in [N]$ such that $f(x) = y$, if one exists. The hardness of function inversion underpins the security of almost every cryptographic primitive we use in practice: block ciphers, hash functions, digital signatures, and so on. Understanding the exact complexity of function inversion is thus critical for assessing the security of our most important cryptosystems.

We are particularly interested in function-inversion algorithms that only make black-box use of the function f—or formally, that have only oracle access to f—since these algorithms invert all functions. A straightforward argument shows that any black-box inversion algorithm that makes at most T queries to its f-oracle succeeds with probability at most O(T/N), over the randomness of the adversary and the random choice of the function. This argument suggests that an attacker running in o(N) time cannot invert a black-box function on domain [N] with good probability.

When the inversion algorithm may use preprocessing, this logic breaks down. An algorithm with preprocessing runs in two phases: In the preprocessing phase, the algorithm repeatedly queries f and then outputs an “advice string” about f. In the subsequent online phase, the algorithm takes as input its preprocessed advice string and a challenge point $y \in [N]$. It must then produce a value $x \in [N]$ such that $f(x) = y$. When using these algorithms for cryptanalysis, the attacker typically seeks to jointly minimize the bit-length S of the advice string and the running time T of the online algorithm. The computation required to construct the advice string, though usually expensive, can often be amortized over a large number of online inversions.

A trivial preprocessing algorithm stores a table of $f^{-1}$ in its entirety as its advice string using $S = \widetilde{O}(N)$ bits and can then invert the function on all points using a single lookup into the table. In contrast, constructing algorithms that simultaneously achieve sublinear advice and online time $S = T = o(N)$ is non-trivial.

In a seminal paper, Hellman [46] introduced time-space tradeoffs as a tool for cryptanalysis and gave a black-box preprocessing algorithm that inverts a function $f{:}\,[N] \rightarrow [N]$ using only $S = \widetilde{O}(N^{2/3})$ bits of advice and online time $T = \widetilde{O}(N^{2/3})$, where the algorithm is guaranteed to succeed only on a constant fraction of functions. (More precisely, the algorithm has a constant success probability over the uniformly random choice of the function f.) Fiat and Naor [27, 28] later gave a rigorous analysis of Hellman’s algorithm and extended it to invert all possible functions, albeit with a slightly worse trade-off of the form $S^3 T= \widetilde{O}(N^3)$ for any choice of ${N}^{3/4} \le S \le N$. Hellman’s trade-off is the best known today, and his algorithm is a fundamental tool in real-world cryptanalysis [7, 8, 59, 61].

In this work, we investigate the following question:

Is it possible to improve upon Hellman’s time-space trade-off?

Yao first asked this question in 1990 [77] and proved that any preprocessing algorithm for function inversion that uses S bits of advice and T online queries must satisfy $ST={{\widetilde{\Omega }}}(N)$. (Counting only queries—and not online computation—only strengthens lower bounds in this model.) Notably, this lower bound does not rule out an algorithm that achieves $S = T = \widetilde{O}(N^{1/2})$. In contrast, Hellman’s algorithm only gives an upper bound of $S = T = \widetilde{O}(N^{2/3})$, even for the slightly easier case of inverting a random function. The question resurfaces in the work of Fiat and Naor [28], Barkan, Biham, and Shamir [5] (who show that Hellman’s method is optimal for a certain natural but restricted class of algorithms), De, Trevisan and Tulsiani [21], and Abusalah et al. [1].

In addition to the problem’s theoretical appeal, determining the best possible time-space trade-offs for function inversion is relevant to practice, since the difference between an online attack time of $N^{2/3}$ and an $N^{1/2}$ becomes crucial when dealing with 128-bit block ciphers, such as the ubiquitous AES-128. Hellman’s algorithm gives the best known preprocessing attack against AES-128, with $S = T \approx 2^{86}$. If we could improve Hellman’s algorithm to achieve $S = T = N^{1/2}$, matching Yao’s lower bound, we could break AES-128 in time $2^{64}$ with a data structure of size $2^{64}$, albeit after an expensive preprocessing phase. While today’s $S=T=2^{86}$ attack is likely far beyond the power of any realistic adversary, an improved $S=T=2^{64}$ attack would leave us with an alarmingly narrow security margin.

Recent work proves new lower bounds on preprocessing algorithms for various cryptographic problems, using both incompressibility arguments [1, 23, 32] and the newer presampling method [19, 65]. While this progress might give hope for an improved lower bound for function inversion as well, both techniques mysteriously fail to break the $ST = {{\widetilde{\Omega }}}(N)$ barrier.

Non-adaptive Algorithms. Another avenue for study is to explore the role of parallelism or adaptivity in preprocessing algorithms for function inversion. All non-trivial algorithms for function inversion, including Hellman’s algorithm and Rainbow-table methods [61], critically use the adaptivity of their queries. It would be very interesting to construct a highly parallelizable preprocessing algorithm for function inversion. Such an algorithm would achieve the same advice and time complexity $S = T = \widetilde{O}(N^{2/3})$ as Hellman’s algorithm, but would make all $\widetilde{O}(N^{2/3})$ of its queries to the f-oracle in one non-adaptive batch. Such a non-adaptive inversion algorithm could speed up function inversion on cryptanalytic machines with a very large number of parallel processing cores.

We do not even know if there exists a non-adaptive algorithm with $S=T=o(N)$. Can we find new non-adaptive inversion algorithms, or is adaptivity necessary for good time-space trade-offs? Proving lower bounds in this more restricted model could be a stepping stone to improving the general lower bounds on function inversion.

1.1 Our Results

This work establishes new connections between the function-inversion problem and well-studied problems in cryptography, complexity theory, and data structures. These connections are useful in two directions.

First, they shed new light on the function-inversion problem: a connection to circuit complexity suggests that improving on the known lower bounds for function-inversion will be difficult. In particular, we show that new lower bounds for function inversion will imply new circuit lower bounds and could even resolve complexity-theoretic questions that predate Hellman’s results [66]. Moreover, a new connection to the problem of breaking PRGs with preprocessing suggests a new avenue for better inversion algorithms for injective functions. For many of the cryptanalytic applications, progress on this variant of function inversion would in fact be sufficient.

Second, these connections, together with classic cryptanalytic algorithms, give rise to better algorithms for problems in the other areas of theory. For example, a connection to communication complexity leads to the best known algorithm for the multiparty pointer-jumping problem, improving upon a twenty-year-old upper bound [63]. Similarly, a connection to data structures leads to a new upper bound for the systematic substring-search problem, resolving an open question [29].

We now state our results in detail.

Proving Better Lower Bounds for Function-Inversion Implies New Circuit Lower Bounds. A major question in circuit complexity, open since the 1970s [66, 67], is to give an explicit family of functions $F_n{:}\,\{0,1\}^n\rightarrow \{0,1\}^n$ that cannot be computed by fan-in-two circuits of size O(n) and depth $O(\log n)$. Following ideas of Brody and Larsen [13], we demonstrate a close connection between this classic problem in circuit complexity and non-adaptive preprocessing algorithms for function inversion.

Specifically, we show that proving that every non-adaptive black-box function-inversion algorithm that uses $S=N\log N/\log \log N$ bits of advice requires at least $T = \Omega (N^{\epsilon })$ oracle queries, for some constant $\epsilon > 0$, would give an explicit family of functions that cannot be computed by linear-size log-depth Boolean circuits. This, in turn, would resolve a long-standing open problem in circuit complexity. Though we cannot prove it, we suspect that the above lower bound holds even for $\epsilon =1$.

This connection implies that proving lower bounds against non-adaptive function-inversion algorithms that use the relatively large amount of advice $S=N\log N/\log \log N$ should be quite difficult. A much more modest goal would be to rule out any non-adaptive algorithm using $S = T = \widetilde{O}(N^{1/2+\epsilon })$, for some $\epsilon > 0$. This would represent only a slight strengthening of Yao’s $ST = {{\widetilde{\Omega }}}(N)$ bound for adaptive algorithms. However, we show that achieving even this far-more-modest goal would improve the best known lower bound for circuits in Valiant’s common-bits model [66, 67]. This, in turn, would represent substantial progress towards proving lower bounds against linear-size log-depth circuits. In particular, since any lower bound against algorithms without a restriction on adaptivity would only be more general, improving the $ST = {{\widetilde{\Omega }}}(N)$ lower bound for function inversion would imply new circuit lower bounds in Valiant’s common-bits model.

We believe that the difficulty of proving such a circuit lower bound suggests that beating the square-root barrier exhibited by both the compression [33, 77] and presampling [19, 65] techniques might prove more difficult than previously expected.

One-to-One Function Inversion from PRG Distinguishers. Many cryptanalytic applications of Hellman tables (cryptanalysis of block ciphers, password cracking, etc.) only require inverting injective functions. Does there exist a better-than-Hellman algorithm for inverting injective functions with preprocessing?

One reason to hope for a better algorithm for injective functions is that for the very special case of permutations, there exists an inversion algorithm with preprocessing that achieves the improved trade-off $ST=\widetilde{O}(N)$ (i.e., $S=T=N^{1/2}$) [77]. Can we achieve the same trade-off for injective functions?

While we have not been able to answer this question yet, we do open one possible route to answering it. In particular, we show that the problem of inverting injective functions with preprocessing has a close connection to the problem of breaking pseudorandom generators (PRGs) with preprocessing [2, 19, 21, 23, 24]. Specifically, De, Trevisan, and Tulsiani [21] show that black-box PRG distinguishers with preprocessing can realize the trade-off $S = \widetilde{O}(\epsilon ^2 N)$, for $T = \widetilde{O}(1)$ and for any choice of distinguishing advantage $\epsilon$.

We show that achieving a more general trade-off of the form $ST = \widetilde{O}(\epsilon ^2 N)$, for any constant $\epsilon$, would imply a better-than-Hellman algorithm for inverting injective functions. Thus, improving the known PRG distinguishers with preprocessing can improve the known injective inversion algorithms.

New Protocols for Multiparty Pointer Jumping. We show that algorithms for the black-box function-inversion problem are useful in designing new communication protocols for a well-studied problem in communication complexity. In particular, any black-box preprocessing algorithm for inverting permutations yields a protocol for the permutation variant of the “k-party pointer-jumping” problem (${\mathsf {MPJ}_{N,k}^\mathsf {perm}}$) [10, 11, 14, 20, 56, 63, 70] in the number-on-the-forehead model of communication complexity [15].

Then, by instantiating the permutation-inversion algorithm with a variant of Hellman’s method, we obtain the best known protocol for ${\mathsf {MPJ}_{N,k}^\mathsf {perm}}$ for $k=\omega (\log N / \log \log N)$ players (this regime is in fact the most consequential for the original motivation for studying this problem), improving the previous best upper bound of $O(N\log \log N/\log N)$, by Pudlák et al. [63], to $\widetilde{O}(N/k+\sqrt{N})$. We thus make progress on understanding the communication complexity of multiparty pointer jumping, a problem with significance to $\mathsf {ACC}^0$ circuit lower bounds [6, 47, 78].

Beyond the quantitative improvement, our protocol is different from all previous approaches to the problem and is an unexpected application of a cryptanalytic algorithm to a communication-complexity problem. While the use of a cryptanalytic algorithm in this context appears new, prior work has found application of results in communication complexity to lower bounds [44] and constructions [12] in the cryptographic setting.

This connection presents a path forward for proving non-adaptive lower bounds for permutation inversion. In particular, we show that for every non-adaptive black-box permutation-inversion algorithm using S bits of advice and T online queries, it must hold that $\max \{S,T\}$ is at least as large as the communication complexity of ${\mathsf {MPJ}_{N,3}^\mathsf {perm}}$. Any improvement on the lower bound for ${\mathsf {MPJ}_{N,3}^\mathsf {perm}}$ would give an improved lower bound for non-adaptive black-box permutation-inversion algorithms. The best lower bound for ${\mathsf {MPJ}_{N,3}^\mathsf {perm}}$ is $\Omega (\sqrt{N})$ [3, 73]. Interestingly, this matches the best lower-bound for black-box permutation-inversion algorithms, regardless of their adaptivity.

New Time-Space Trade-Off for Systematic Substring Search. Finally, we show that improved algorithms for function inversion will also imply improved data structures for the systematic substring-search problem [22, 29, 30, 40, 41]. In particular, we prove that there is a preprocessing algorithm for the function-inversion problem using few bits of advice and few online queries if and only if there is a space- and time-efficient data structure for systematic substring search in the cell-probe model [75]. In the systematic substring-search problem, we are given a bitstring of length N (the “text”), and from it we must construct an S-bit data structure (the “index”). Given a query string, we should be able to determine whether the query string appears as a substring of the text by reading the index and by inspecting at most T bits of the original text.

This connection is fruitful in two directions: First, we show that instantiating this connection with the Fiat-Naor algorithm for function inversion [28] yields an $S^3T=\widetilde{O}(N^3)$ systematic data structure, which is the best known in the parameter regime $S =\widetilde{O}(N^\alpha )$ for $\alpha < 1$. Gál and Miltersen [29] ask whether a very strong $S + T = {{\widetilde{\Omega }}}(N)$ lower bound on this problem is possible. By beating this hypothetical lower bound, our algorithm answers their open question in the negative.

Second, Gál and Miltersen prove an $ST = {{\widetilde{\Omega }}}(N)$ lower bound for systematic substring search. Our barrier to proving lower bounds against black-box algorithms for function inversion implies that improving this lower bound would also imply new lower bounds in Valiant’s circuit model and therefore may be quite challenging.

1.2 Related Work

We now recall a few salient related results on function inversion, and we discuss additional related work at relevant points throughout the text.

Fiat and Naor [27, 28] proved that Hellman’s algorithm [46] achieves a trade-off of the form $S^2T=\widetilde{O}(N^2)$, when the algorithm needs only to invert a random function with constant probability (i.e., in the cryptanalytically interesting case). For the worst-case problem of inverting arbitrary functions, Fiat and Naor give an algorithm that achieves a trade-off of the form $S^3T = O(N^3)$. De, Trevisan, and Tulsiani [21] improve the Fiat-Naor trade-off when the algorithm needs only to invert the function at a sub-constant fraction of points.

For inverting functions, Yao [77] proved that every algorithm that uses S bits of advice and makes T online queries must satisfy $ST = {{\widetilde{\Omega }}}(N)$ lower bound. Impagliazzo gives a short alternative proof [48]. Dodis et al. [23], building on prior work [21, 33], extended the lower bound to capture algorithms that invert only a sub-constant fraction of functions f.

Barkan, Biham, and Shamir [5] show that, for a restricted class of preprocessing algorithms, a Hellman-style trade-off of the form $S^2 T = \widetilde{O}(N^2)$ is the best possible. Their lower bound is powerful enough to capture the known inversion schemes, including Hellman’s algorithm and Oechslin’s practically efficient “Rainbow tables” technique [61]. At the same time, this restricted lower bound leaves open the possibility that an entirely new type of algorithm could subvert their lower bound.

For inverting permutations, Yao [77] observed that a Hellman-style algorithm can achieve the $ST = \widetilde{O}(N)$ upper bound and proved a matching lower bound. Gennaro and Trevisan [33], Wee [71], and De, Trevisan, and Tulsiani [21] extend this lower bound to handle randomized algorithms and those that succeed with small probability.

Two recent works [39, 52] use the function-inversion algorithm of Fiat and Naor to obtain new algorithms for the preprocessing version of the 3-SUM problem.

1.3 Preliminaries

Notation. Through this paper, $\mathbb {Z}^{\ge 0}$ denotes the non-negative integers, and $\mathbb {Z}^{> 0}$ denotes the positive integers. For any $N\in \mathbb {Z}^{>0}$ we write $[N] = \{1, 2, \dots , N\}$. We often identify every element $x\in [N]$ with the binary representation of $x-1$ in $\{0,1\}^{\lceil \log N\rceil }$. We use $x \leftarrow 4$ to denote assignment and, for a finite set $\mathcal {X}$, we use to denote a uniform random draw from $\mathcal {X}$. For a function $f:A\rightarrow B$ and $y\in B$, we define the preimage set of y as $f^{-1}(y)\mathrel {\mathop :}=\{x\in A\mid f(x)=y\}$. All logarithms are base-two unless stated otherwise. Parameters S and T are always implicit functions of the parameter N, and to simplify the bounds, we always implicitly take $S = T = \Omega (1)$. The notation ${{\widetilde{\Omega }}}(\cdot )$ and $\widetilde{O}(\cdot )$ hides factors polynomial in $\log N$.

Definition 1

(Black-box inversion algorithm with preprocessing). Let $N \in \mathbb {Z}^{>0}$. A black-box inversion algorithm with preprocessing for functions on [N] is a pair $(\mathcal {A}_0,\mathcal {A}_1)$ of oracle algorithms, such that $\mathcal {A}_0$ gets oracle access to a function $f{:}\,[N]\rightarrow [N]$, takes no input, and outputs an advice string $\textsf {st}_f\in \{0,1\}^*$. Algorithm $\mathcal {A}_1$ gets oracle access to a function $f{:}\,[N]\rightarrow [N]$, takes as input a string $\mathsf {st}_f \in \{0,1\}^*$ and a point $y\in [N]$, and outputs a point $x\in [N]$. Moreover, for every $x\in [N]$, it holds that $\mathcal {A}_1^f(\mathcal {A}_0^f(),f(x)) \in f^{-1}(f(x))$.

We can define a black-box inversion algorithm for permutations analogously by restricting the oracle $f{:}\,[N]\rightarrow [N]$ to implement an injective function. In this case, we will often denote the oracle as $\pi$ instead of f.

Definition 2

(Adaptivity). We say that an oracle algorithm is k-round adaptive if the algorithm’s oracle queries consist of k sets, such that each set of queries depends on the advice string, the input, and the replies to the previous rounds of queries. We call a 1-round adaptive algorithm non-adaptive. Finally, we say that an algorithm is strongly non-adaptive if it issues a single set of queries that only depends on the algorithm’s input, but not on the advice string. In all of the above cases, when referring to the number of queries made by the algorithm, we account for the sum over all rounds.

Worst Case Versus Average Case. The algorithms in Definition 1 are deterministic and successfully invert all functions on all points. It is also interesting to consider algorithms that invert successfully only with probability $\epsilon <1$, over the random choice of: the function $f{:}\,[N] \rightarrow [N]$, the point to invert, and/or algorithm’s randomness. As most of the results in this paper deal with barriers for improving lower bounds, restricting ourselves to deterministic algorithms that always succeed in inverting only makes these results stronger. In any case, assume that all algorithms we consider halt with probability 1.

Running Time Versus Query Complexity. For the purposes of proving lower bounds, and reductions towards proving lower bounds, it suffices to consider the query complexity of a preprocessing algorithm’s online phase. Counting only queries (and not computation time) only strengthens lower bounds proved in this model. The algorithms we construct can be made to use only $\widetilde{O}(N)$ preprocessing time in a suitable RAM model, when they are allowed to fail with small probability. Furthermore, the running time of our algorithms’ T-query online phase is $\widetilde{O}(T)$.

Non-uniformity. Our definition allows for “free” non-uniformity in the parameter N. Nevertheless, in a model that only “charges” the online algorithm for queries to the oracle and ignores the actual running time, non-uniformity makes little difference since a uniform algorithm can simply search for the optimal choice of non-uniform advice without increasing its query complexity.

Shared Randomness. We allow the preprocessing and online phases to access a common stream of random bits. Allowing the adversary to access correlated randomness in both phases only strengthens the lower bounds. Only one of our upper bounds (Theorem 8) makes use of this correlated randomness.

2 Lower Bounds on Inversion Imply Circuit Lower Bounds

The motivating question of this work is whether Hellman’s $S = T = \widetilde{O}(N^{2/3})$ algorithm for inverting random functions is optimal. In this section, we show that resolving this question will require proving significant new lower bounds in Valiant’s “common bits” model of circuits [66]. We also show that proving strong lower bounds on non-adaptive algorithms for function inversion would imply new lower bounds against linear-sized logarithmic-depth circuits.

We obtain these connections by observing that the function-inversion problem is an example of a class of so called “succinct” static data-structure problems [4, 17, 30, 31, 40, 41, 43, 45, 49, 58, 64]. We show a barrier to proving lower bounds against systematic data structures, which are a special case of succinct data structures.

Related Work. Brody and Larsen [13] showed that proving certain lower bounds against linear data structures for dynamic problems would imply strong lower bounds on the wire complexity of linear depth-two circuits. We follow their general blueprint, but we instead focus on arbitrary algorithms for solving static data-structure problems (e.g., function inversion), and our connection is to Valiant’s common-bits model of circuits, rather than to linear depth-two circuits.

In recent independent work, Viola [69, Theorem 3] shows that lower bounds against a large class of static data-structures problems imply circuit lower bounds. In his work, Viola considers an incomparable circuit model that, on the one hand, admits circuits of depth larger than two, but, on the other hand, restricts the number of wires connected to the common bits. As a result, Viola’s work does not seem to apply to the function-inversion problem within the relevant parameter regime (namely, in the gap between Hellman’s upper bound and Yao’s lower bound).

In another recent independent work, Dvir, Golovnev, and Weinstein [26] connect data-structure lower bounds to matrix rigidity and circuit lower bounds. Their focus is on linear data structures, whereas the function inversion problem, considered in our work, does not have an apparent linear structure.

Boyle and Naor [9] make a surprising connection between cryptographic algorithms and circuit lower bounds. They show that proving the non-existence of certain “offline” oblivious RAM algorithms (ORAMs) [34, 38, 62] would imply new lower bounds on the size of Boolean circuits for sorting lists of integers. Larsen and Nielsen [54] recently skirted this barrier by proving a lower bound against ORAMs in the “online” setting. Following that, Weiss and Wichs [72] showed that a variant of the Boyle-Naor barrier still holds against “online read-only” ORAMs.

2.1 Systematic Data Structures and Low-Depth Circuits

A major open question in circuit complexity is whether there exists an explicit family of Boolean functions (from n bits to one bit) that cannot be computed by fan-in-two circuits of size O(n) and depth $O(\log n)$. An easier problem, which is still famously difficult, is to find an explicit family of functions $F_n{:}\,\{0,1\}^n\rightarrow \{0,1\}^n$ with n-bit output—often called Boolean operators—that cannot be computed by this same class of circuits. Even this question has been open since the 1970s [51, 66, 67].

More precisely, we say that a family of Boolean operators $\{ F_n \}_{n\in \mathbb {Z}^{>0}}$, for $F_n{:}\,\{0,1\}^n\rightarrow \{0,1\}^n$, is an explicit operator if the decision problem associated with each bit of the output of $F_n$ is in the complexity class $\mathsf {NP}$.

The main result of this section is that proving a certain type of data-structure lower bound implies the existence of an explicit Boolean operator on n bits that cannot be computed by fan-in-two circuits of size O(n) and depth $O(\log n)$. We then show that a lower bound on function-inversion algorithms can be cast as a data-structure lower bound, and therefore a function-inversion lower bound implies a circuit lower bound.

We now give the necessary background on data-structure problems. A systematic data structure of size s and query complexity t for an operator $F_n$ is a pair of algorithms:

• a preprocessing algorithm, which takes as input the data $x\in \{0,1\}^n$ and outputs a string $\mathsf {st}\in \{0,1\}^s$ of length $s = o(n)$, and

• a query algorithm, which takes as input the string $\mathsf {st}$, and an index $i\in [n]$, may probe (read) t bits of the input x, and then outputs the ith bit of $F_n(x)$.

A systematic data structure is non-adaptive if the query algorithm probes a set of bits of the input data x whose location depends only on the index i and not on the input data x.

The following theorem is the main result of this section.

Theorem 3

If an explicit operator $\{F_n\}_{n\in \mathbb {Z}^{>0}}$ has fan-in-two Boolean circuits of size O(n) and depth $O(\log n)$ then, for every $\epsilon > 0$, then this operator admits a non-adaptive systematic data structure of size $O(n/ \log \log n)$ and query complexity $O(n^{\epsilon })$.

To prove this, we first recall Valiant’s common-bits model of circuits [66, 67].

Fig. 1.

Common-bits circuit with $n=4$ inputs, degree $d=2$, and width $w=2$.

Valiant’s Common-Bits Model. A circuit in the common-bits model of width w and degree d computing a Boolean operator $F_n{:}\,\{0,1\}^n\rightarrow \{0,1\}^n$ contains an input layer, a middle layer, and output layer (Fig. 1). The input layer consists of n input bits $x_1, \dots , x_n \in \{0,1\}$, and the output layer consists of n output gates. There are w gates in the middle layer of the circuit (the “common bits”); each input feeds into each of these w middle gates, and the output of each of the w middle gates feeds into each output gate. Further, each output gate reads from at most d of the inputs. Unlike in a standard circuit, the gates in the middle and output layers of the circuit compute arbitrary functions of their inputs. The output of the circuit is the n-bit string formed at the output gates.

It is immediate that any Boolean operator $F_n{:}\,\{0,1\}^n\rightarrow \{0,1\}^n$ has common-bits circuits of width n and degree 0 or, alternatively, of width 0 and degree n. A non-trivial question is: For a given operator $F_n$ and choice of degree (e.g., $d=n^{1/3}$), what is minimal width of a common-bits circuit that computes $F_n$?

Lemma 4

If there exists a circuit in the common-bits model of width w and degree d that computes an operator $F{:}\,\{0,1\}^n\rightarrow \{0,1\}^n$, then there exists a non-adaptive systematic data structure for F of size w and query complexity d.

Proof

Let $\mathcal {C}$ be a circuit in the common-bits model as in the statement of the lemma. The data structure consists of the outputs of the w middle-layer gates in the circuit $\mathcal {C}$ (i.e., the circuit’s common bits). On input $i\in [n]$, the algorithm reads all the input bits connected to the ith output gate of $\mathcal {C}$ and computes the value of the output gate. Since each output gate in the circuit is connected to at most d input bits, the query complexity of the systematic data structure is at most d.   $\square$

Theorem 3 then follows from Lemma 4 and the following result of Valiant:

Theorem 5

(Valiant [66, 67]). If every explicit operator has fan-in-two Boolean circuits of size O(n) and depth $O(\log n)$, then for every constant $\epsilon >0$, every explicit operator has circuits in the common-bits model of width $O(n/\log \log n)$ and degree $n^{\epsilon }$.

Viola [68, Section 3] and Jukna [50, Chapter 13] give detailed proofs of Theorem 5.

2.2 Consequences for Function Inversion

Observe that every function $f{:}\,[N] \rightarrow [N]$ can be described using $O(N \log N)$ bits, so there is a trivial strongly non-adaptive algorithm that inverts every function using $O(N \log N)$ bits of advice and no queries to the function in the online phase. We know of no non-adaptive function-inversion algorithm that inverts with constant probability using $o(N \log N)$ bits of advice and o(N) queries. The following theorem states that ruling out the existence of such a non-adaptive algorithm is as hard as proving lower bounds against linear-size logarithmic-depth Boolean circuits.

Theorem 6

If, for some $\epsilon > 0$, every family of strongly non-adaptive black-box algorithms for inverting functions $f{:}\,[N] \rightarrow [N]$ that uses $O(N^{\epsilon })$ queries requires $\omega (N \log N / \log \log N)$ bits of advice, then there exists an explicit operator that cannot be computed by fan-in-two Boolean circuits of size O(n) and depth $O(\log n)$.

The theorem considers a restricted class of inversion algorithms that: (i) may only use strongly non-adaptive queries (the most restrictive type of query), (ii) are only allowed, for example, $O(N^{0.0001})$ queries (very few queries), and (iii) must invert arbitrary functions with probability one (the most difficult variant of the inversion problem).

So, even though we may suspect that there are no algorithms for inverting functions $f{:}\,[N] \rightarrow [N]$ using $O(N \log N /\log \log N)$ bits of advice and $O(N^{0.0001})$ non-adaptive queries, proving such an assertion seems very challenging.

Proof

of Theorem 6. Let $n = N \log N$, where $N \in \mathbb {Z}^{>0}$ is a power of two. (For all other values of n, define the inversion operator trivially as the identity mapping.) We define the inversion operator $F^{\mathsf {inv}}_n{:}\,\{0,1\}^n\rightarrow \{0,1\}^n$ as follows. Let $x\in \{0,1\}^n$ be an input to $F^{\mathsf {inv}}_n$, and view x as the concatenation of N blocks of length $\log N$ bits each: $x=x_1 \Vert x_2 \Vert \cdots \Vert x_{N}$. For each $i \in [N]$, let $y_i \in [N]$ be the least $j \in [N]$ such that $x_j = i$, if one such j exists. If no such j exists, set $y_i = 0$. We define $F^{\mathsf {inv}}_n(x)= \big (y_1 \Vert y_2 \Vert \cdots \Vert y_N\big )$.

Observe that a systematic data structure for $F^{\mathsf {inv}}_n$ gives a strongly non-adaptive preprocessing algorithm that inverts every function $f{:}\,[N] \rightarrow [N]$. The preprocessing phase constructs the data structure for operator $F^{\mathsf {inv}}_n$ on input $f(1) \Vert f(2) \Vert \dotsc \Vert f(N)$ and outputs this data structure as the advice string.

In the online phase, on input $i\in [N]$, the algorithm uses the data structure in the advice string and its oracle access to f to compute all $\log N$ bits of the ith output block $y_i$ of $F^{\mathsf {inv}}_n$, which is enough to recover some inverse of i under f, if it exists.

The theorem now follows from Theorem 3, instantiated with $F^{\mathsf {inv}}_n$. For n of the form $n = N \log N$, where $N > 0$ is a power of two, we get that the length of the advice string is $O(N \log N / \log \log (N \log N)) = O(N \log N / \log \log N)$ and the online query complexity is $\log N\cdot O\left( (N\log N)^\epsilon \right) = O(N^{\epsilon '})$, for any $\epsilon ' > \epsilon$.    $\square$

Theorem 6 suggests the hardness of proving stronger lower bounds for non-adaptive inversion algorithms, but it applies only to algorithms that use a relatively long advice string, of length $O(N\log N/\log \log N)$. We might still hope to improve upon Yao’s $ST={{\widetilde{\Omega }}}(N)$ lower bound for function inversion without breaking the aforementioned barrier.

The following corollary shows that ruling out function-inversion algorithms using advice and time $S = T = \widetilde{O}(N^{1/2 + \epsilon })$, for any $\epsilon > 0$, would imply the existence of an explicit operator that cannot be computed by circuits of width $O(n^{1/2 + \epsilon '})$ and degree $O(n^{1/2+ \epsilon '})$ in the common-bits model, for some $\epsilon ' > 0$. As we will discuss, no such lower bound in the common-bits model is known, so proving the optimality of Hellman’s $\widetilde{O}(N^{2/3})$ algorithm, or even showing that inverting functions with preprocessing is marginally harder than inverting permutations with preprocessing, would imply an advance in the state of lower bounds on circuits in the common-bits model.

Corollary 7

If, for some $\epsilon > 0$, there does not exist a family of strongly non-adaptive algorithms for inverting functions $f{:}\,[N] \rightarrow [N]$ using $O(N^{1/2 + \epsilon })$ bits of advice and $O(N^{1/2 + \epsilon })$ queries, then there exists an explicit operator that does not have circuits in the common-bits model of width $O(n^{1/2 + \epsilon '})$ and degree $O(n^{1/2 + \epsilon '})$, for every $\epsilon '$ satisfying $0< \epsilon ' < \epsilon$.

Proof

We prove the contrapositive. Assume that for every $\epsilon ' > 0$, every explicit operator has common-bits circuits of width $O(n^{1/2 + \epsilon '})$ and depth $O(n^{1/2 + \epsilon '})$. Then, as in the proof of Theorem 6, we can apply Lemma 4 to operator $F^{\mathsf {inv}}_n$ to show that, for $n = N \log N$, there exists a strongly non-adaptive preprocessing algorithm that inverts functions $f{:}\,[N] \rightarrow [N]$ using $O(n^{1/2 + \epsilon '}) = O((N \log N)^{1/2 + \epsilon '}) = O(N^{1/2 + \epsilon '} \log N)$ bits of advice and $O(n^{1/2 + \epsilon '} \log N) = O((N \log N)^{1/2 + \epsilon '} \log N)$ online queries. Then, for any $\epsilon > \epsilon '$, the advice usage and number of online queries is $O(N^{1/2+\epsilon })$.   $\square$

Notice that while the hypothesis of Corollary 7 considers a lower bound against strongly non-adaptive inversion algorithms, this only strengthens the statement. This is true because proving a lower bound against adaptive inversion algorithms implies a lower bound against strongly non-adaptive algorithms as well.

If we instantiate Corollary 7 with $\epsilon = 1/6$, we find that ruling out function-inversion algorithms using $S = T = o(N^{2/3})$, even against the restricted class of strongly non-adaptive algorithms, would give an explicit operator that does not have common-bits circuits of width w and degree d satisfying $w = d = o(n^{2/3 - \delta })$, for any $\delta > 0$.

Proving such a lower bound on common-bits circuits is not strong enough to yield a lower bound against linear-size log-depth circuits via Valiant’s method (Theorem 5). However, this lower bound would improve the best known lower bound against circuits in the common-bits model. The best known bound, due to Pudlák, Rödl, and Sgall, gives $d=\Omega (\frac{n}{w}\cdot \log (\frac{n}{w}))$, for a common-bits circuit of width w and degree d [63]. In particular, they construct an explicit operator that does not have common-bits circuits satisfying $w = d = \widetilde{O}(n^{1/2})$. By Corollary 7, ruling out function-inversion algorithms with $S = T = \widetilde{O}(N^{1/2+\epsilon })$, for any $\epsilon > 0$, would thus improve the best lower bounds on common-bits circuits.

2.3 Consequences for Other Succinct Data-Structure Problems

Theorem 3 and Lemma 4 together imply that proving strong lower bounds for any systematic data-structure problem—not only for the function-inversion problem—will be challenging. To explain how this barrier applies to a completely different data-structure problem, we recall the systematic variant of the standard data-structure problem of polynomial evaluation with preprocessing [57]. We give an informal description of the problem, and the transformation into a formal systematic data-structure problem (as in Sect. 2.1) is straightforward.

The problem of polynomial evaluation with preprocessing is parameterized by an integer $N \in \mathbb {Z}^{>0}$ and a finite field $\mathbb {F}$ of size $\Theta (N)$. The input data is a polynomial $p \in \mathbb {F}[X]$ of degree at most $N-1$, represented as its vector of coefficients $\bar{c} = (c_0, c_1, \dots , c_{N-1}) \in \mathbb {F}^N$. The preprocessing algorithm reads this input (the entire polynomial p) and produces a preprocessed S-bit string $\mathsf {st}$. In a subsequent online phase, the query algorithm takes as input a point $x_0 \in \mathbb {F}$, and must output the evaluation $p(x_0) \in \mathbb {F}$ of the polynomial p at point $x_0$. To produce its answer, the query algorithm may read the entire preprocessed string $\mathsf {st}$, query at most T coordinates of the coefficient vector $\bar{c}$, and perform an unlimited amount of computation.

For what choices of space usage S and query complexity T does there exist a systematic data structure for polynomial evaluation with preprocessing?

The two naïve approaches to solving this problem are:

• 1. Have the preprocessing algorithm store in the string $\mathsf {st}$ the evaluation of the polynomial p on every point in the field $\mathbb {F}$, using $S = \Omega (N)$ space.

• 2. Have the online-phase algorithm read the entire coefficient vector $\bar{c}$, using $T = \Omega (N)$ queries, and then evaluate $p(x_0)=\sum _i c_i x_0 \in \mathbb {F}$ directly.

These solutions both have $S + T = {{\widetilde{\Omega }}}(N)$.

It seems very difficult to construct an algorithm that simultaneously uses a data structure of size $S = N^\delta$ and query complexity $T = N^\delta$, for some $\delta < 1$. And yet, the best lower bound we have for this problem, implied by a bound of Gál and Miltersen [30], is of the form $ST = {{\widetilde{\Omega }}}(N)$. A variant of Corollary 7 implies that proving stronger lower bounds for this problem—or proving any lower bound better than $ST = {{\widetilde{\Omega }}}(N)$ for any systematic or succinct data-structure problem, for that matter—will also imply new lower bounds in Valiant’s common-bits model. Proving even a stronger lower bound could, via Theorem 3, imply a lower bound against linear-size log-depth fan-in-two circuits.

3 Breaking PRGs Is as Hard as Inverting Injective Functions

Many cryptanalytic applications of Hellman tables only require inverting injective functions. That is given a injective function $f:[N]\rightarrow [M]$ and a point $y\in [M]$, find a value $x\in [N]$ such that $f(x)=y$, if one exists.

For example, consider the classic application of Hellman tables to plaintext attacks on block ciphers: Let $E:\{0,1\}^k\times \{0,1\}^n\rightarrow \{0,1\}^n$ be a block cipher, where k is the key size and n is the block size. If we define $f_E:\{0,1\}^k\rightarrow \{0,1\}^n$ such that $f_E(x)=E(x,m_0)$ for some fixed plaintext $m_0$, then an algorithm with preprocessing for the function $f_E$ essentially gives a known-plaintext attack on the block cipher E. We can (heuristically) expect the resulting function $f_E$ to behave similar to a random function, and therefore be injective only beyond the birthday bound $k\gtrapprox 2n$. However, even for shorter keys, we can reduce a known-plaintext attack to the problem of inverting an injective function by considering the encryption of multiple known plaintexts $m_0, m_1, m_2$. For example, if $k=n$, then we expect $f_{E^{\times 3}}{:}\,\{0,1\}^n\rightarrow \{0,1\}^{3n}$, defined as $f_{E^{\times 3}}(x)=E(x,m_0)\Vert E(x,m_1)\Vert E(x,m_2)$, to have no collisions.

A function-inversion algorithm can invert an injective function $f{:}\,[N] \rightarrow [M]$ without taking any advantage of the fact that it is injective, so Hellman’s $S^2 T = \widetilde{O}(N^2)$ upper bound for function inversion [46] applies in this setting as well. However, the fact that for the case of random permutations (i.e., an injective function $f:[N]\rightarrow [N]$), Hellman’s algorithm gives a significantly better upper bound of $ST=\widetilde{O}(N)$, gives hope that a similar improvement—or at least some improvement—is possible for injective length-increasing functions.

To the best of our knowledge, the injective variant of the function-inversion problem has not been studied directly so far, even though it is a special case with wide cryptanalytic applications. As a first step, we connect the injective inversion problem to the problem of breaking pseudorandom generators (PRGs) with preprocessing [2, 19, 21, 23, 24]. In that problem, we model a “black-box” PRG as an oracle $G{:}\,[N] \rightarrow [M]$, with $N < M$. A PRG distinguisher with preprocessing first makes arbitrarily many queries to G and outputs an S-bit advice string. In the online phase, the distinguisher can then use its advice string, along with T queries to G, to distinguish whether a given sample $y\in [M]$ has been drawn from the distribution or the distribution .

In their work, De, Trevisan, and Tulsiani [21] give a distinguisher with $S=O(\epsilon ^2 N)$ and $T=\widetilde{O}(1)$ that achieves a distinguishing advantage $\epsilon \le 1/\sqrt{N}$. They ask whether it is possible to realize the trade-off $ST = \widetilde{O}(\epsilon ^2 N)$ for other parameter settings as well. The following theorem shows that a PRG distinguisher that achieves constant distinguishing advantage at points on this trade-off (e.g., $\epsilon = 1/100$, $S = N^{1/4}$, and $T = N^{3/4}$) would imply a better-than-Hellman algorithm for inverting injective functions.

Theorem 8

Suppose that there is a black-box PRG distinguisher that uses S bits of advice, makes T online queries to a PRG $G{:}\,[N]\rightarrow [M]$, and achieves distinguishing advantage $\epsilon$. Then there exists a black-box algorithm that inverts any injective function $f{:}\,[N]\rightarrow [M]$ using $\widetilde{O}(\epsilon ^{-2} S)$ bits of advice and $\widetilde{O}(\epsilon ^{-2} T)$ online queries, and that inverts f with probability $1-1/\log N$ (over the algorithm’s randomness).

Furthermore, if the preprocessing and online phase algorithms have access to a common random oracle, the online phase also runs in time $\widetilde{O}(T)$.

Remark 9

(Relation to Goldreich-Levin). A classic line of results [36, 37, 55, 76] shows how to use any injective one-way function $f: [N] \rightarrow [M]$ to construct an efficient PRG $G_f: [N^2] \rightarrow [2N^2]$ which makes black-box use of f. The proof uses the Goldreich-Levin theorem [37] to show that any efficient distinguisher for $G_f$ yields an inversion algorithm for f. (Consult Goldreich’s textbook [35, Section 3.5] for the details.) It is not clear to us whether a non-uniform generalization of these classic results directly implies Theorem 8. The problem is that the domain of the PRG $G_f$ has size $N^2$, whereas the domain of the original function f has size N. Since we are interested in the exact exponent of the advice and time usage of function-inversion algorithms (i.e., $S = N^{3/4}$ versus $S = N^{1/2}$), we are sensitive to this polynomial expansion in the domain size. For example, say that we were able to construct a black-box PRG distinguisher that achieves $S = T = \widetilde{O}(\sqrt{N})$. Applying the classic reduction directly to $G_f$ would only imply the existence of an inverter for the function f that uses the trivial advice and time complexity $S = T = \widetilde{O}(\sqrt{N^2})=\widetilde{O}(N)$. In contrast, Theorem 8 implies that an $S = T = \widetilde{O}(\sqrt{N})$ distinguisher yields an $S = T = \widetilde{O}(\sqrt{N})$ inverter.

Proof

Idea for Theorem 8. Given a distinguisher for any length-increasing generator $G{:}\,[N]\rightarrow [M]$, we construct an inversion algorithm for injective functions ${f{:}\,[N]\rightarrow [M]}$ in two steps. First, for each $i \in [n]$, we construct a bit-recovery algorithm $\mathcal {B}_i$ that, given f(x), achieves a non-trivial advantage in recovering the $i{\text {th}}$ bit of x. We then use the algorithms $(\mathcal {B}_1, \dots , \mathcal {B}_n)$ to construct an inversion algorithm $\mathcal I$ that, given f(x), recovers the full preimage x with good probability.

To give the intuition behind the bit-recovery algorithm $\mathcal {B}_i$: Given a function $f{:}\,[N] \rightarrow [M]$ to invert, we construct a function $G_i{:}\,[N] \rightarrow [M]$ such that a point $y = f(x)$ is in the image of $G_i$ if and only if the $i{\text {th}}$ bit of x is 1. Then, we can apply the PRG distinguisher to $G_i$ and recover the $i{\text {th}}$ bit of y’s preimage.

This simple algorithm does not quite work when the PRG distinguisher has small distinguishing advantage $\epsilon$, since the distinguisher may fail on the point y. To fix this, we give $\mathcal {B}_i$ access to two random permutations $\pi {:}\,[N] \rightarrow [N]$ and $\sigma {:}\,[M] \rightarrow [M]$ that allow $\mathcal {B}_i$ to essentially randomize the point it gives as input to the PRG distinguisher.

We then can run $\mathcal {B}_i$ many times with different random permutations and then take the majority vote of the outputs of these runs. This majority vote will yield the $i{\text {th}}$ bit of the x with high probability. To complete the construction, we instantiate the permutations $\pi$ and $\sigma$ using correlated randomness between the preprocessing and online algorithms. The full description of the construction appears in the full version of this paper.   $\square$

4 From Cryptanalysis to New Communication Protocols

Communication complexity [53, 74] quantifies the number of bits that a set of players need to communicate amongst themselves in order to compute a function on an input that is split between the players. One of the major open problems in communication complexity is to obtain a non-trivial lower bound for some problem for a super-poly-logarithmic number of players. Such a bound would in turn lead to a breakthrough circuit lower bound for the complexity class $\mathsf {ACC}^0$ [6, 47, 78].

In this section, we develop connections between the function-inversion problem and the multiparty pointer-jumping problem in the number-on-the-forehead (NOF) model of communication complexity [15]. By combining these new connections with the classic cycle-walking algorithm for permutation inversion, we obtain the best known NOF protocols for the permutation variant of the pointer-jumping problem. Since pointer jumping is a candidate hard problem in the k-party NOF setting, understanding the exact communication complexity of pointer jumping for a super-poly-logarithmic number of players is an important step towards the eventual goal of proving circuit lower bounds [10, 11, 14, 20, 56, 63, 70].

4.1 Multiparty Pointer-Jumping in the NOF Model

A classical problem in the NOF model is the pointer-jumping problem. We describe the permutation variant of the problem, and then discuss the general case. In the pointer-jumping problem ${\mathsf {MPJ}_{N,k}^\mathsf {perm}}$, there are k computationally-unbounded players, denoted $P_0,P_1,\dotsc , P_{k-1}$, and each has an input “written on her forehead.” The first player $P_0$ has a point $x\in [N]$ written on her forehead, the last player $P_{k-1}$ has a Boolean mapping $\beta : [N] \rightarrow \{0,1\}$ written on her forehead, and each remaining player $P_i$, for $i=1,\dots ,k-2$, has a permutation $\pi _i:[N]\rightarrow [N]$ written on her forehead. Each player can see all $k-1$ inputs except the one written on her own forehead. The goal of the players is to compute the value $\beta \circ \pi _{k-2}\circ \cdots \circ \pi _1(x)$, which loosely corresponds to “following a trail of pointers” defined by the permutations, starting from x (Fig. 2). The players can communicate by writing messages on a public blackboard. The communication complexity of a protocol is the total number of bits written on the blackboard for a worst-case input.

A one-way protocol is a protocol in which each player writes a single message on the blackboard in the fixed order $P_0,\dotsc , P_{k-1}$, and the last player’s message must be the output. The one-way communication complexity of a function f, denoted $\mathsf {CC}^1(f)$, is the minimum communication complexity of all one-way protocols that successfully compute f. Without the “one-way” restriction, there are protocols for ${\mathsf {MPJ}_{N,k}^\mathsf {perm}}$ that require only $O(\log N)$ bits of communication.

Known Bounds. The best upper bound for ${\mathsf {MPJ}_{N,k}^\mathsf {perm}}$ is due to Pudlák et al. [63], who showed that $\mathsf {CC}^1({\mathsf {MPJ}_{N,k}^\mathsf {perm}}) = O(N\log \log N / \log N)$. More recently, Brody and Sanchez [14] showed that this upper bound applies to the more general pointer-jumping problem, in which we replace the permutations $\pi _1, \dots , \pi _{k-2}$ with arbitrary functions. In this general case, Wigderson [73] proved an $\Omega (\sqrt{N})$ lower bound for $k=3$ players (see also [3]), and Viola and Wigderson [70] proved an ${{\widetilde{\Omega }}}(N^{\frac{1}{k-1}})$ lower bound for $k \ge 3$ players.

Fig. 2.

A pointer-jumping instance for ${\widehat{\mathsf {MPJ}}_{k=4,N=5}^\mathsf {perm}}$ with $\pi _1=(1\, 2\, 4\, 5\, 3)$, $\pi _2=(2\, 3)(4\, 5)$, $\pi _3=(2\, 3\, 4\, 5)$ and $x=2$. Lemma 11 reduces this instance to inverting the permutation $\pi _1^{-1}\pi _2^{-1}\pi _3^{-1}=(1\, 3\, 5\, 4)$ on the point $x=2$.

4.2 A New Communication Protocol from Permutation Inversion

We obtain the best known communication protocol for the permutation variant of the pointer-jumping game on parameter N for $k = \omega (\log N / \log \log N)$ players. Our result improves the previously best known upper bound of $\widetilde{O}(N)$ to $\widetilde{O}\big (N/k + \sqrt{N}\big )$. Extending our upper bound to the general multiparty pointer-jumping problem remains an open problem, which we discuss in Remark 13.

On the lower-bound side, this connection suggests a path to prove lower bounds against partially adaptive permutation-inversion algorithms, as in Definition 2. In contrast, the techniques of Sect. 2 can only prove lower bounds against strongly non-adaptive algorithms.

In this section, we prove the following new upper bound on $\mathsf {CC}^1({\mathsf {MPJ}_{N,k}^\mathsf {perm}})$:

Theorem 10

$\mathsf {CC}^1({\mathsf {MPJ}_{N,k}^\mathsf {perm}}) \le O\left( (N/k+\sqrt{N})\log N\right)$.

To prove Theorem 10, as we do later in this section, we use the integer-valued version the pointer-jumping problem, commonly denoted ${{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}$. In this version, the last player $P_{k-1}$ holds a permutation $\pi _{k-1}: [N] \rightarrow [N]$, instead of a boolean mapping, so the output of the problem is a value in [N]. The following technical lemma, which we prove in Appendix A, shows that the boolean-valued version of the pointer-jumping problem has communication complexity that is not much larger than that of the integer-valued pointer-jumping problem.

Lemma 11

$\mathsf {CC}^1({\mathsf {MPJ}_{N,k}^\mathsf {perm}}) \le \mathsf {CC}^1({{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}) + \lceil \log N \rceil \,.$

Then, our main lemma technical uses an arbitrary permutation-inversion algorithm with preprocessing to solve the integer-valued problem ${{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}$:

Lemma 12

If there exists a $(k-2)$-round adaptive algorithm for inverting permutations $\pi : [N] \rightarrow [N]$ that uses advice S and time T, then

$$\mathsf {CC}^1({{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}})\le S + T \lceil \log N \rceil \,.$$

Proof

Let $(\mathcal {A}_0,\mathcal {A}_1)$ be a $(k-2)$-round adaptive algorithm for inverting permutations with preprocessing. We give a protocol for ${{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}$.

• Player $P_0$ runs the preprocessing algorithm $\mathcal {A}_0$ on the permutation $\pi ^{-1}_1\circ \cdots \circ \pi ^{-1}_{k-1}$ and writes the advice string on the blackboard.

• Player $P_1$ runs the online inversion algorithm $\mathcal {A}_1$ on the input x (written on player $P_0$’s forehead) using the advice string that has been written on the blackboard, to produce the first round of queries $q_{1,1},\dotsc ,q_{1,t_1}$. For each query $q_{1,\ell }$, she computes the partial reply $p_{1,\ell }=\pi ^{-1}_2(\dots (\pi ^{-1}_{k-1}(q_{1,\ell }))\dots )$ and writes it on the blackboard.

• Player $P_i$, for $i \in \{2, \dots , k-2\}$, reads the partial replies $p_{i-1,1},\dotsc ,p_{i-1,t_{i-1}}$ written by the previous player, computes the (complete) query replies $r_{i-1,1},\dotsc ,r_{i-1,t_{i-1}}$ by computing $r_{i-1,\ell }=\pi ^{-1}_1(\dotsc (\pi ^{-1}_{i-1}(p_{i-1,\ell }))\dotsc )$. Player $P_i$ then runs (in her head) the first $i-1$ rounds of the online inversion algorithm on input x, using the advice string and the replies to the first $i-1$ rounds of queries, all of which, she can compute using the partial replies written on the blackboard. Player $P_i$ then produces the $i^{\text {th}}$ round of queries, on which, similarly to Player $P_1$, she computes the partial replies and writes them on the blackboard.

• Player $P_{k-1}$ completes the evaluation of round $k-2$ of the queries by evaluating the remaining permutations $\pi ^{-1}_1\circ \cdots \circ \pi ^{-1}_{k-2}$ on the partial replies written by $P_{k-2}$. Player $P_{k-1}$ then runs in her head all $k-2$ rounds of the online inversion algorithm and writes the output on the blackboard.

By definition, the output y of the algorithm satisfies $\pi ^{-1}_1\circ \cdots \circ \pi ^{-1}_{k-1}(y)=x$. Since all $\pi _i$ are permutations, it must hold $\pi _{k-1}\circ \cdots \circ \pi _{1}(x)=y$ and so y is the correct output for ${{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}$.

The communication consists of the advice string written by Player $P_0$ and a partial reply for each query, giving a total of $S+T \lceil \log N \rceil$. (The last player writes the $\lceil \log N \rceil$-bit output, but does not need to write the response to the T-th query).   $\square$

Proof

of Theorem 10. To prove Theorem 10, we instantiate Lemma 12 using Hellman’s cycle-walking algorithm [46], which we recall in the full version of this paper. The algorithm inverts permutations using T queries and S bits of advice, for every choice of S and T such that $ST \ge 2N \lceil \log N + 1\rceil$. Furthermore the algorithm is T-round adaptive. Specifically, for $k \le \sqrt{N}+2$, using Hellman’s algorithm with $T=k-2$ and $S=\lceil (2N\log N) / T \rceil$ gives a protocol with communication $O( (N/k)\log N)$. For $k > \sqrt{N} + 2$, we use Hellman’s algorithm with $T=\sqrt{N}$ and $S=2 \sqrt{N} (\lceil \log N \rceil + 1)$ to find that $\mathsf {CC}^1({{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}) \le O(\sqrt{N} \log N)$. Then, applying Lemma 11 lets us conclude that $\mathsf {CC}^1({\mathsf {MPJ}_{N,k}^\mathsf {perm}}) \le O(\sqrt{N} \log N)$.   $\square$

Remark 13

(The function case). We might hope to show that a good function-inversion algorithm, such as that of Fiat and Naor [28], implies a good protocol for the general multiparty pointer-jumping problem, in which each player i has an arbitrary function $f_i$ (which may not be a permutation) written on her forehead. We do not know how to prove such a result. The problem is that the reduction of Lemma 12 requires that the composition $f^{-1}_1\circ f^{-1}_2\circ \cdots \circ f^{-1}_{k-1}$ is a function, and this is not true in the general case. (In contrast, when $f_1,\dots ,f_k$ are all permutations it holds that $f^{-1}_1\circ f^{-1}_2\circ \cdots \circ f^{-1}_{k-1}$ is a permutation.) Since several upper bounds for the permutation variant of the pointer-jumping problem [11, 20, 63] have led to subsequent upper bounds for the unrestricted case [11, 14], there is still hope to generalize the result.

5 From Cryptanalysis to Data-Structures

In this section, we show how to apply the Fiat-Naor algorithm for function inversion [28] to obtain the best known data structure for the systematic substring-search problem [22, 29, 30, 40, 41], in a wide range of parameter regimes. As a consequence of this connection, we show that the open problem of improving the known lower bounds on function inversion is equivalent to the open problem in the data-structure literature of whether it is possible to improve the known lower bounds for systematic substring search.

In the systematic substring-search problem, we are given a bitstring of length N (“the text”) and a bitstring of length $P \ll N$ (“the pattern”). If the pattern appears in the text, we must output an index $i \in [N]$ into the text at which the pattern begins. We take the pattern length to be $P = \Theta (\log N)$.

An algorithm for systematic substring search is a two-part algorithm $\mathcal {A}= (\mathcal {A}_0,\mathcal {A}_1)$. The preprocessing algorithm $\mathcal {A}_0$ takes as input only the text, may perform arbitrary computation on it, and then outputs an S-bit “index” into the text. The online algorithm $\mathcal {A}_1$ takes as input the index and the pattern, queries T bits of the text, and then outputs the location of pattern in the text, if one exists.

By applying the Fiat-Naor function inversion algorithm [28], we obtain the best known algorithm for systematic substring search on texts of length N when using an index of size $O(N^\epsilon )$ bits, for any $\epsilon < 1$. Gál and Miltersen [30] asked for a strong lower bound against search algorithms using an $O(N/{{\text {polylog}}}\,N)$-bit index, and we answer this question by giving an upper bound that beats their hypothetical lower bound. This connection also gives evidence that finding a faster algorithm for systematic substring search will require a cryptanalytic breakthrough.

Known Lower Bounds. Demaine and López-Ortiz [22] prove that on texts of length N with pattern length $P = \Theta (\log N)$, any algorithm that uses an S-bit index and makes $T = o(P^2/\log P)$ queries in the online phase must satisfy $ST = \Omega (N \log N)$. Golynski [40, 41] gives a stronger version of this bound that applies even for larger $T = o(\sqrt{N}/\log N)$. Gál and Miltersen prove a slightly weaker bound but that holds for all values of T. They show that for certain pattern lengths $P = \Theta (\log N)$, and any choice of T, any algorithm must satisfy $ST = \Omega (N/ \log N)$.¹

The main technical result of this section is the following theorem, which we prove in the full version of this paper.

Theorem 14

For any integer $N \in \mathbb {Z}^{>0}$ and integral constant $c > 2$, if there is an algorithm for systematic substring search on texts of length $cN\cdot \lceil \log N \rceil$ with pattern length $c \cdot \lceil \log N \rceil$ that uses an S-bit index and reads T bits of the text in its online phase, then there is a black-box algorithm for inverting functions $f{:}\,[N] \rightarrow [N]$ that uses S bits of advice and makes T online queries.

For any integer $N \in \mathbb {Z}^{>0}$, if there is a black-box algorithm for inverting functions $f{:}\,[2N] \rightarrow [2N]$ that uses S bits of advice and T queries, then, for any integral constant $c > 1$, there is an algorithm for systematic substring search on texts of length N with pattern length $c \cdot \lceil \log N \rceil$ that uses an $\widetilde{O}(S)$-bit index and reads $\widetilde{O}(T)$ bits of the text in its online phase.

Remark 15

It is possible to make the preprocessing time $\widetilde{O}(N)$ by allowing the algorithm to fail with probability O(1/N) over the randomness of the preprocessing phase. Similarly, the online running time (in addition to the query complexity) is $\widetilde{O}(T)$.

Proof

Idea. The full proof appears in the full version of this paper. In the first part, we must use a substring-search algorithm to invert a function $f{:}\,[N] \rightarrow [N]$. The idea is to construct a text $\tau$ of length $\Theta (N \log N)$ by writing out the evaluation of f at all points in its domain, in order, with a few extra bits added as delimiters. To invert a point $y \in [N]$, we use the substring search algorithm to find the location at which y appears in the text $\tau$. This location immediately yields a preimage of y under f. Demaine and López-Ortiz [22] use a similar—but more sophisticated encoding—on the way to proving a data-structure lower bound for systematic substring search. Their encoding maps a function $f{:}\,[N] \rightarrow [N]$ into a string of length $(1 + o(1)) N \log N$, while ours maps f into a string of length $3 N \log N$.

In the second part, we must use a function-inversion algorithm to solve substring search on a text $\tau$ of length N with pattern length $P = c \cdot \lceil \log N \rceil$, for some constant $c > 1$. To do so, we define a function $f'{:}\,[N] \rightarrow [N^c]$ such that $f'(i)$ is equal to the length-P substring that starts from the $i{\text {th}}$ bit of the text $\tau$. Given a pattern string $\sigma = \{0,1\}^P$, finding the inverse of y under $f'$ is enough to locate the position of the pattern string $\sigma$ in the text $\tau$. The only remaining challenge is that $f'$ is length-increasing, rather than length-preserving. We then use universal hashing to reduce the problem of inverting length-increasing functions to the problem of inverting length-preserving functions, which completes the proof.   $\square$

We now apply Theorem 14 to construct a new algorithm for systematic substring search that resolves an open question of Gál and Miltersen. In their 2007 paper, Gál and Miltersen say that “it would be nice to prove a lower bound of, say, the form,” $T < N/{{\text {polylog}}}\, N \Rightarrow S > N/{{\text {polylog}}}\, N$ (using our notation) for systematic substring search [30]. Goyal and Saks [42] use an elegant argument to show that the specific technique of Gál and Miltersen cannot prove this lower bound. As a corollary of Theorem 14, we construct an algorithm for substring search that beats the hypothetical lower bound.

Corollary 16

For any integral constant $c > 1$ there is an algorithm for systematic substring search on texts of length N with pattern length $c \cdot \lceil \log N \rceil$, that uses an S-bit index, reads T bits of the text in its online phase, and achieves the trade-off $S^3 T = \widetilde{O}(N^3)$.

Proof

Theorem 14 shows that systematic substring search on strings of length N with pattern length $\Theta (\log N)$ reduces to the problem of inverting arbitrary functions $f{:}\,[N] \rightarrow [N]$. The inversion algorithm of Fiat and Naor [28] inverts such functions f achieving the desired complexity bounds.    $\square$

In particular, we get an algorithm that solves systematic substring search using an index size and time satisfying $S = T = \widetilde{O}(N^{3/4})$, for strings of length N and patterns of length $\Theta (\log N)$. Furthermore, this connection, along with the results of Sect. 2.2, shows that improving on the $ST = {{\widetilde{\Omega }}}(N)$ bound of Gál and Miltersen will require advances in techniques for proving lower bounds on the power of depth-two circuits.

6 Discussion and Future Directions

In this final section, we discuss a few directions for future work.

6.1 Which Lower-Bound Techniques Can Work?

In Sect. 2, we showed that improving Yao’s lower bound on function-inversion algorithms requires new circuit lower bounds in the common-bits model. What potential approaches do we have to prove such a lower bound?

Function Inversion and Yao’s “Box Problem.” Yao’s “box problem” [60, 77] is a preprocessing problem that is closely related to the function-inversion problem. In the box problem, we are given oracle access to a function $f{:}\,[N] \rightarrow \{0,1\}$. First, we get to look at all of f and write down an S-bit advice string $\mathsf {st}_f$. Later on, we are given our advice string $\mathsf {st}_f$ and a point $x \in [N]$. We may then make T queries to f, provided that we do not query f(x), and we must then output a value $y \in \{0,1\}$ such that $y = f(x)$.

The box problem is in some sense the dual of the function-inversion problem: we are given an f-oracle and we must compute f in the forward direction, rather than in the inverse direction. The same $ST = {{\widetilde{\Omega }}}(N)$ lower bound applies to both problems [77]. However, in contrast to the inversion problem, for which we suspect that good parallel (i.e., non-adaptive) algorithms do not exist, the natural algorithm for the box problem is already non-adaptive and achieves $ST = O(N \log N)$.²

Puzzlingly, the two main techniques for proving time-space lower bounds do not distinguish between the function-inversion problem and Yao’s box problem. In particular, the known lower bounds use compression [21, 23, 33, 77] or bit-fixing [18, 19, 65]. Both techniques essentially look at the information that the oracle queries and their replies give on the pair (x, f(x)) induced by the challenge, regardless of whether the actual challenge is x, and the algorithm has to find f(x) (as in the case of Yao’s box problem), or the challenge is $y=f(x)$, and the algorithm has to find $x=f^{-1}(y)$ (as in the case of the inversion problem).

Since there is an $ST = \widetilde{O}(N)$ upper bound for Yao’s box problem, then any method that proves a lower bound better than $ST = \Omega (N)$ for function inversion must not apply to the box problem. Therefore, a “sanity check” for any improved lower bound for the function-inversion problem is to verify that the same proof technique does not apply to Yao’s box problem.

Strong-Multiscale-Entropy. Drucker [25] shows, at the very least, that improving lower bounds in the common-bits model will require new types of arguments. In particular, Jukna [50, Chapter 13], generalizing earlier arguments of Cherukhin [16] defined the “strong multiscale entropy” (SME) property of Boolean Operators. Jukna proved that an operator on n bits with the SME property cannot be computed by common-bits circuits of width $o(n^{1/2})$ and degree $o(n^{1/2})$. (These results are actually phrased in terms of the wire complexity of depth-two circuits with arbitrary gates, but the implications to the common-bits model are straightforward.)

Strengthening Jukna’s lower bound on the circuit complexity of SME operators appeared to be one promising direction for progress on lower bounds. Thwarting this hope, Drucker constructs an explicit operator with the SME property that has circuits in the common-bits model of width $O(n^{1/2})$ and degree $O(n^{1/2})$. Thus, SME-type arguments alone are not strong enough to prove that an operator cannot be computed by circuits of width $O(n^{1/2 + \epsilon })$ and degree $O(n^{1/2+\epsilon })$ for $\epsilon > 0$.

6.2 One-to-One Functions

Prompted by the fact that many cryptanalytic applications of function inversion only require inverting injective function, we initiated in Sect. 3 the study of injective function inversion. Though we take the first step by connecting this problem to the problem of distinguishing PRGs, the basic question remains: is it easier to invert a random injective function $f{:}\,[N] \rightarrow [M]$, for $N \ll M$, than it is to invert a random length-preserving function $f{:}\,[N] \rightarrow [N]$? A better-than-Hellman attack against injective functions would be remarkable. Or, can we prove that inverting injective functions is as hard as inverting random functions?

6.3 Barriers for Upper Bounds

Is there a barrier to getting an $S = T = o(N^{2/3})$ algorithm for function inversion? Barkan, Biham, and Shamir [5] prove a lower bound against a certain restricted class of Hellman-like algorithms, which suggests that better algorithms must use new techniques. It would be satisfying to show at least that improving Hellman’s upper bound would result in a dramatic algorithmic improvement for a well-studied problem in another domain.

A Proof of Lemma 11

The first step is to define a permutation $\pi _\beta {:}\,[N] \rightarrow [N]$, for every function $\beta {:}\,[N] \rightarrow \{0,1\}$. We then use this permutation $\pi _\beta$ to convert a Boolean-valued pointer-jumping instance $(x, \pi _1, \dots , \pi _{k-1}, \beta )$ to an integer-valued pointer-jumping instance $(x, \pi _1, \dots , \pi _{k-1}, \pi _\beta )$. Solving the integer-valued instance using a protocol for ${{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}$ is then enough—with a few extra bits of communication—to solve the Boolean-valued instance of ${\mathsf {MPJ}_{N,k}^\mathsf {perm}}$.

Table 1. Example of the encoding procedure of Lemma 11. $N=2^3$, and $\beta {:}\,[N] \rightarrow \{0,1\}$. Note that the last column is a permutation over the elements of [N]. Also note how $\beta$ can be recovered from $\pi _\beta (x)$ for all $x\ne 0$.

Towards constructing $\pi _\beta$, consider first the case when N is a power of two. For $N=2^n$, consider the following mapping from $\{0,1\}^N$ to permutations on $\{0,1,\dotsc ,N-1\}= \{0,1\}^n$. On $\beta {:}\,\{0,1\}^n \rightarrow \{0,1\}$ we construct a permutation $\pi _\beta$ on $\{0,1\}^n$ as follows: let $x\in \{0,1\}^n$ and let $x=x_nx_{n-1}\dotsc x_1$ be the binary representation of x. Set $\pi _\beta (x)=y=y_ny_{n-1}\dotsc y_1$ defined by $y_i=\beta ( x|_i)\oplus x_i \oplus 1$ where $x|_i=0\dotsc 01x_{i-1}x_{i-2}\dots x_1$. The following two properties hold:

• The mapping $\pi _\beta$ defined above is a permutation. To see this let $x\ne x'$ be two distinct elements in $\{0,1\}^n$, and let $y=\pi _\beta (x)$ and $y'=\pi _\beta (x')$. Let $i\in [n]$ be the rightmost bit position on which x and $x'$ differ. Then $x_i \ne x'_i$ but $x|_i = x'|_i$. Therefore $y_i=\beta (x|_i)\oplus x_i \oplus 1 \ne \beta (x'|_i)\oplus x'_i \oplus 1= y'_i$, so $y\ne y'$.

• For any $x\in \{0,1\}^n$ such that $x=x_n\dotsc x_1\ne 0$, let i be the leftmost bit position such that $x_i=1$. It then holds that $\beta (x)$ is equal to the $i{\text {th}}$ bit of $\pi _\beta (x)$.

Note that the latter property guarantees that the value of $\beta (x)$ for every $x\ne 0$ can be recovered from a single bit of $\pi _\beta (x)$.

For N which is not a power of 2, we can view N as a sum $\sum _{j=1}^\ell 2^{n_j}$ of at most $\lceil \log N\rceil$ powers of 2, and construct a permutation $\pi _\beta$ on $\{0,\dotsc ,N-1\}= \{0,1\}^{n_1}\cup \cdots \cup \{0,1\}^{n_\ell }$ as a union of permutations on $\{0,1\}^{n_j}$. By the properties above, for all but $\ell =\lceil \log N \rceil$ bad points, the value of $\beta$ can be recovered from the corresponding value of $\pi _\beta$. Note that the set of bad points depends only on N and not on $\beta$. We give an example of this encoding procedure in Table 1.

Therefore, given a communication protocol for ${{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}$, we construct a protocol for ${\mathsf {MPJ}_{N,k}^\mathsf {perm}}$ as follows. Let $\beta \in \{0,1\}^N$ be the input (on the forehead) of the last player. Each of the first $k-1$ players computes the permutation $\pi _\beta$ from $\beta$ according to the mapping above. The first player also writes on the blackboard the value of $\beta$ evaluated on all of the bad points of $\pi _{\beta }$. The players then run the protocol for ${{\widehat{\mathsf {MPJ}}_{N,k}^\mathsf {perm}}}$ on the instance $(x,\pi _1,\dotsc ,\pi _{k-2}, \pi _{\beta })$.

The last player computes the output of the original protocol $\pi _{\beta }\circ \pi _{k-2} \circ \cdots \circ \pi _1 (x) = \pi _{\beta }(\hat{x}) \in \{0,1,\dotsc ,N-1\}$ where $\hat{x} = \pi _{k-2} \circ \cdots \circ \pi _1 (x)$. If $\hat{x}$ is not a bad point she can recover and output $\beta \circ \pi _{k-2} \circ \cdots \circ \pi _1 (x) = \beta (\hat{x}) \in \{0,1\}$ from $\pi _{\beta }(\hat{x})$. Otherwise, if $\hat{x}$ is a bad point, she outputs the value $\beta (\hat{x})$, which the first player wrote on the blackboard.

The new protocol increases the communication complexity of the original protocol by $\lceil \log N\rceil$.