Word Sense Disambiguation (WSD) systems have recently achieved unprecedented high performance and have approached or even exceeded the human level. However, almost all these systems only focus on integrating knowledge and designing the WSD classification layer while having few works related to loss function, which is equally essential for WSD systems. In this paper, we propose Enhanced Bi-encoder Model (EBEM) with a more flexible loss function, Circle loss, and enhanced representations via knowledge from documents and WordNet. EBEM is strong in fine-grained sense discrimination. Our experimental results show that EBEM produces a considerable performance improvement compared with Bi-encoder Model (BEM) and achieves state-of-the-art results on English all-words WSD. More importantly, EBEM has an excellent performance in few-shot learning, which will contribute to the further application and extension of WSD. We also verify the effectiveness of Circle loss in WSD systems and report how it affects the clusters of sense representations.