A multi-user heterogeneous semantic and bit communication framework is proposed. An access point (AP) serves multiple users via either the semantic communication (SemCom) or the conventional bit communication (BitCom). Orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA) techniques are employed for efficiently facilitating this heterogeneous multi-user network. A long-term (equivalent) semantic throughput maximization problem is formulated for jointly optimizing the communication mode selection, NOMA user pairing scheme, and the corresponding resource allocation at each time slot. A twin-delayed deep deterministic policy gradient with reference neuron enhanced Softmax (TD3-RNS) algorithm is proposed, where a fixed-value neuron is employed as the reference to improve the training efficiency. Simulation results demonstrate that 1) the proposed NOMA-based heterogeneous transmission outperforms the baselines which merely use SemCom or BitCom; 2) the proposed TD3-RNS achieves superior performance than the conventional TD3 algorithm.