Rapid and accurate seismic source characterization significantly influences the performance of earthquake early warning (EEW) systems. However, the complexity of the seismic source modeling and the error accumulation during continuous characterization make it difficult to accurately characterize various source parameters. Furthermore, current artificial intelligence methods focus on a single task, lacking intertask fusion and guidance from specialized knowledge. In this study, we propose a novel deep learning (DL) algorithm (SeisParaNet) to estimate P-wave arrival time, source location, and magnitude simultaneously based on a multitask framework. To exploit seismological knowledge and attenuate the strong intertask dependencies, this study incorporates arrival time differences’ information into the analysis of source localization parameters using the attention mechanism and incorporates source location features into estimating local magnitude (Ml). In addition, SeisParaNet uses a probability-based self-attention mechanism (Prob-Attention) to extract temporal information from waveforms. Experimental results demonstrate that following a limited number of trainings on the STanford EArthquake Dataset (STEAD), SeisParaNet exhibits the capability to capture complex seismic patterns and rapidly characterize seismic sources. Furthermore, the introduction of Prob-Attention reduces computational complexity by 67%, validating the potential of SeisParaNet in EEW applications.