As deep learning neural networks (DNNs) advance and increase in computational complexity, particularly in terms of memory cost, it becomes difficult to implement DNNs in fixed-point memory-sparse environments (e.g. integrated circuits in consumer electronics). Thus, the training of DNNs must be reformulated to balance the hardware costs needed to represent the often millions of parameter weights in such machine learning models. This paper proposes a novel optimization approach that simultaneously minimizes complexity (total memory) and maximizes accuracy. Specifically, a bit-depth complexity penalty is induced to urge the DNN model towards a state of lower memory using a numerical gradient in optimization iterations. Experimental results on the MNIST (Mixed National Institute of Standards and Technology) handwritten digit classification benchmark demonstrate a minimal loss (1%) in DNN accuracy with a significant reduction (37%) of memory cost on average.