Energy management systems (EMS) in smart grids provide end users with the optimal operational efficiency of power from nonsmart microgrids, including power grids, energy storage systems (ESS), and residential loads. This article proposes a novel distributed online control policy for Ambient Intelligence (AmI)-based Internet of Things (IoT) environments, optimizing a consensus utility function, including electricity cost and the lifespan of ESS. Different from the existing methods, the distributed EMS via IoT can gain cooperative $\boldsymbol {L_{2}}$ performance by rejecting external disturbances and providing consensus policies for robust optimal charging and discharging. First, consensus dynamics of AmI-agents are constructed, and the Hamilton–Jacobi-Isaacs (HJI) equations are established, where the Nash equilibrium points are approximated by ADP and zero-sum game theory. Second, with the aid of an actor-critic structure, a robust optimal distributed control algorithm in an online manner for EMS is proposed. Therefore, collecting sample sets and training offline are completely avoided. Third, to deal with the unknown internal dynamics of ESS, the $Q$ -learning algorithm is employed instead of system identification techniques that require available sample sets. The algorithm guarantees that the global load is balanced and that the consensus tracking error and the function approximation error are uniformly ultimately bounded. Finally, numerical simulations are provided to verify the effectiveness of the proposed algorithm for a large-scale system of nonsmart microgrids.