We introduce a recursive scheme for performing Compressed Sensing (CS) on streaming data and analyze, both analytically and experimentally, the computational complexity and estimation error. The approach consists of sampling the input stream recursively via overlapping windowing and making use of the previous measurement in obtaining the next one. The signal estimate from the previous window is utilized in order to achieve faster convergence in an iterative optimization algorithm to decode the new window. To remove the bias of the estimator a two-step estimation procedure is proposed comprising support set detection and signal amplitude estimation. Estimation accuracy is enhanced by averaging estimates obtained from overlapping windows. The proposed method is shown to have asymptotic computational complexity O(nm^3/2), where n is the window length, and m is the number of samples. The variance of normalized estimation error is shown to asymptotically go to 0 if k = O(n^1−∈) as n increases. The simulation results show speed up of at least ten times with respect to applying traditional CS on a stream of data while obtaining significantly lower reconstruction error under mild conditions on the signal magnitudes and the noise level.