Acoustic echo cancellation is a critical component of modern communication systems, facilitating clear conversations between speakers in diverse acoustic environments during telecommunication. The primary objective of echo cancellation is to eliminate the other party’s voice, which is transmitted from speakers to microphones. This challenge involves solving the inverse problem of predicting distorted echoes based on noisy observations. While various approaches have been proposed to tackle this problem, evaluating their performance using recorded data often relies on subjective assessments due to the lack of true reference values. This study focuses on techniques to obtain reliable scores even in referenceless scenarios. Our experimental results demonstrate a higher correlation between the proposed metric and echo attenuation compared to commonly used referenceless metrics. Additionally, this paper explores the conditions under which the evaluation of the proposed metric yields meaningful results. It investigates the relationship between the proposed metric and various degradations observed in echoes. AECSQI brings the distinct advantage of offering scores on a more diverse scale compared to traditional mean opinion scores. This characteristic enables a broader and more comprehensive analysis of the evaluated systems from various perspectives.