The design of deterministic measurement matrices has been the focus of research in compressed sensing from the early stages. In particular, structured measurement matrices are of great interest as they could be efficiently stored. Our focus in this letter is on Toeplitz structure, which naturally arises in linear shift-invariant systems (convolution operator). We design complex-valued Toeplitz matrices with unit modulus elements that have small coherence. The complex phase of the matrix elements are determined by certain polynomials. We provide upper bounds for the coherence of the resulting matrix using tools from analytic number theory, namely, the Weyl sum theorem. Simulation results confirm that the proposed matrices perform similar to the Gaussian Toeplitz matrices of the same size.