We analyze the MX/G/1 queue in the framework of Markov decision processes (MDPs). The service times become known upon arrival, and each job incurs a cost according to a given cost function. The value function is a central concept in MDP theory as it characterizes the value of the system's state with respect to future developments. We derive compact expressions for the generating functions for general families of value functions corresponding to often used cost structures defined in terms of waiting and sojourn times. Moreover, we consider systems with and without setup delays.