Loading [a11y]/accessibility-menu.js
Unbiased discourse segmentation evaluation | IEEE Conference Publication | IEEE Xplore

Unbiased discourse segmentation evaluation


Abstract:

In this paper, we show that the performance measures Pk and Window Diff, commonly used for discourse, topic, and story segmentation evaluation, are biased in favor of seg...Show More

Abstract:

In this paper, we show that the performance measures Pk and Window Diff, commonly used for discourse, topic, and story segmentation evaluation, are biased in favor of segmentations with fewer or adjacent segment boundaries. By analytical and empirical means, we show how this results in a failure to penalize substantially defective segmentations. Our novel unbiased measure k-κ corrects this, providing a single score that accounts for chance agreement. We also propose additional statistics that may be used to characterize important properties of segmentations such as boundary clumping. We go on to replicate a recent spoken-language topic segmentation experiment, drawing conclusions that are substantially different from previous studies concerning the effectiveness of state-of-the-art topic segmentation algorithms.
Date of Conference: 12-15 December 2010
Date Added to IEEE Xplore: 24 January 2011
ISBN Information:
Conference Location: Berkeley, CA, USA

References

References is not available for this document.