Previous work on rate-distortion optimized streaming of light fields assumes that the scheduling task is performed by a central server. To scale to large numbers of simultaneous users, however, it is necessary to shift the computational burden of image scheduling to the clients. In this paper, we present a receiver-driven streaming framework where the packet scheduling is performed at the client. One key challenge of receiver-based streaming is in efficiently transmitting the rate-distortion preamble necessary for the scheduling algorithm. We present a method to efficiently represent and estimate the view-dependent distortions, and show that the performance is nearly as good as if the actual distortion values were used. We also show that the sender-driven and receiver-driven systems have very similar rate-distortion streaming performance.