Dedicated neural-network inference-processors improve latency and power of the computing devices. They use custom memory hierarchies that take into account the flow of operators present in neural networks and convolutional layers. For efficient implementation, such network topologies can greatly benefit from hardware-cost optimization using automated network-architecture search. Thereby, cost functions predict the suitability of a network topology for a given type of inference hardware. A differentiable neural-architecture search that optimizes both weights and topology in a single training requires cost models to be differentiable in the dimensions of weight and activation matrices. State-of-the-art differentiable cost models require time-consuming system-level measurements or simulation results, or do not encounter the hardware structure at all. This work presents a simple yet effective procedure for deriving a differentiable neural-network-accelerator cost-model that is suitable for any type of accelerator. It is based on hardware-independent parameterization and a novel differentiable divide-ceil function, as well as hardware-specific modeling. The resulting differentiable model can be reconfigured to the actual hardware size and memory structure to predict the inference energy for an exact network topology. The modeling and prediction are demonstrated for a state-of-the-art SRAM-based inference-accelerator and for the Eyeriss accelerator, inferring different state-of-the-art neural networks, resulting in excellent agreement with measured hardware.