Meeting the diverse quality-of-service (QoS) requirements in ultra-dense Internet of Things (IoT) networks operating under varying network loads is challenging. Moreover, latency-critical IoT applications cannot afford excessive control signaling overheads caused by centralized access control methods. A distributed network access approach can potentially address this problem. In this regard, multi-agent multi-armed bandit (MAB) learning is a promising tool for designing distributed network access protocols. This paper proposes a multi-agent MAB learning-based grant-free access mechanism for ultra-dense networks, where multiple base stations (BSs) serve massive delay-sensitive and delay-tolerant IoT devices. Delay-sensitive devices are prioritized to choose the BSs with larger numbers of channels in a probabilistic manner. The proposed mechanism enables the devices to improve their BS selection over time to accommodate the maximum number of devices that can meet a prescribed latency-reliability criterion. Simulation results show that the proposed MAB learning-based network access mechanism outperforms the random BS selection strategy in which end devices do not employ any learning scheme to adapt to the network dynamics.