Traditionally, video encoders have been designed assuming that the more redundancy is removed, the better the encoder. However, on current laptops, reducing the compression efficiency of the video encoder by reducing the number of instructions used to perform compression can actually reduce the total energy used to encode and transmit a sequence. The correct balance between computation and compression efficiency may change dynamically, motivating adaptive encoders. At the same time, recent general-purpose processors also employ energy-driven adaptations. For best gains, the adaptations in the hardware and application layers must be coordinated. From a system design viewpoint, this coordination must happen through minimal, well-defined interfaces. This paper develops (1) an adaptive video encoder for general-purpose processors that trades computational complexity for compression efficiency to minimize total system energy, and (2) a method for determining the best configuration for such an encoder when running on a processor that is also adaptive. Our adaptive processor employs recent energy saving techniques of dynamic voltage and frequency scaling and architectural adaptation. Using a detailed simulator, we show that our cross-layer adaptive application algorithm reduces energy significantly, when employed on a fixed or adaptive processor.