In smart factories, a variety of automated guided vehicles (AGVs) communicate with cyber-physical systems (CPSs) to autonomously deliver raw materials and workpieces among smart production facilities. In practice, instead of acquiring more costly AGVs to cause congestion in existing working space, most factories develop rule-based and model-based approaches to improve the AGV utilization rate and further the production efficiency. However, these charging strategies require predefined rules or models for estimating the internal information of batteries, so that they lead to huge computational costs and estimation errors. As a consequence, this work creates a Markov decision process problem for real-time charging scheduling of AGVs to fulfill uncertain AGV dispatching requests from the CPS for production lines, in which four bounds for charging heterogeneous AGVs are considered from practical experiences for increasing the AGV utilization rate. This work further improves a feature-based reinforcement learning approach, in which the state and action space can be effectively reduced through approximating the state-value function by five feature functions, including the estimated revenue for improving the utilization time, the total AGV charging cost, the cost of penalizing unfulfilled dispatching requests, the priority of charging newer batteries, and the priority of charging the batteries close to be fully charged, respectively. Experimental results show that the proposed algorithm obtains better benefits than the current practical approach, and improves the AGV utilization rate.