As a distributed machine learning model, federated learning ensures the legitimate use of data and user privacy while training the global model. Existing privacy protection mechanisms for federated learning either need to balance training accuracy and privacy protection requirements, or lack of design from the perspective of groups. In this paper, we design a privacy protection mechanism from the perspective of groups for federated learning. By resorting to cryptographic techniques, the proposed mechanism is free of the tradeoff between accuracy and privacy. In particular, we aim to develop novel approaches for the asymmetric group key agreement (AGKA) protocol with efficient operations and lower storage cost, as well as to further support anonymous group authentication. First, we propose a BLS-AGKA protocol by using the Boneh-Lynn-Shacham (BLS) signature, which is computationally efficient and requires a relatively small storage cost. Second, to further achieve the privacy-preserving demand in federated learning, we construct an anonymous authentication scheme based on the proposed BLS-AGKA protocol, which supports anonymous group authentication. Finally, it is shown that the proposed protocol and scheme guarantee the desired security properties, including session-key security, unforgeability, and anonymity. In addition, the performance of the proposed scheme is superior to relevant existing works as well.