Article

One-sided measures for evaluating ranked retrieval effectiveness with spontaneous conversational speech

Authors:

SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 673 - 674

https://doi.org/10.1145/1148170.1148311

Published: 06 August 2006 Publication History

Get Access

Abstract

Early speech retrieval experiments focused on news broadcasts, for which adequate Automatic Speech Recognition (ASR) accuracy could be obtained. Like newspapers, news broadcasts are a manually selected and arranged set of stories. Evaluation designs reflected that, using known story boundaries as a basis for evaluation. Substantial advances in ASR accuracy now make it possible to build search systems for some types of spontaneous conversational speech, but present evaluation designs continue to rely on known topic boundaries that are no longer well matched to the nature of the materials. We propose a new class of measures for speech retrieval based on manual annotation of points at which a user with specific topical interests would wish replay to begin.

References

[1]

Arons, B., SpeechSkimmer: A System for Interactively Skimming Recorded Speech, ACM TOCHI, 4(1)3--98, 1997.

Digital Library

Google Scholar

[2]

Garofolo, J. et al., The TREC Spoken Document Retrieval Track: A Success Story, in TREC-8, 2000.

Google Scholar

[3]

Gustman, S. et al., Supporting Access to Large Digital Oral History Archives, in JCDL 2002, pp.18--27.

Digital Library

Google Scholar

[4]

Kekäläinen, J. et al., Using Graded Relevance Assessments in IR evaluation, JASIST, 53(13), pp.1120--1129.

Digital Library

Google Scholar

[5]

Voorhees, E., Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, in SIGIR 1998.

Digital Library

Google Scholar

Cited By

View all

Skorkovská L(2016)Relevant Documents Selection for Blind Relevance Feedback in Speech Information RetrievalText, Speech, and Dialogue10.1007/978-3-319-45510-5_48(418-425)Online publication date: 3-Sep-2016
https://doi.org/10.1007/978-3-319-45510-5_48
Skorkovská L(2016)Comparison of Retrieval Approaches and Blind Relevance Feedback Methods Within the Czech Speech Information RetrievalSpeech and Computer10.1007/978-3-319-43958-7_21(182-190)Online publication date: 13-Aug-2016
https://doi.org/10.1007/978-3-319-43958-7_21
Ward NWerner SGarcia FSanchis E(2015)A prosody-based vector-space model of dialog activity for information retrievalSpeech Communication10.1016/j.specom.2015.01.00468:C(85-96)Online publication date: 1-Apr-2015
https://dl.acm.org/doi/10.1016/j.specom.2015.01.004
Show More Cited By

Index Terms

One-sided measures for evaluating ranked retrieval effectiveness with spontaneous conversational speech
1. Information systems
  1. Information retrieval
  2. Information storage systems

Recommendations

Combining LVCSR and vocabulary-independent ranked utterance retrieval for robust speech search
SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval

Well tuned Large-Vocabulary Continuous Speech Recognition (LVCSR) has been shown to generally be more effective than vocabulary-independent techniques for ranked retrieval of spoken content when one or the other approach is used alone. Tuning LVCSR ...
Characterizing and detecting spontaneous speech: Application to speaker role recognition

Processing spontaneous speech is one of the many challenges that automatic speech recognition systems have to deal with. The main characteristics of this kind of speech are disfluencies (filled pause, repetition, false start, etc.) and many studies have ...
Detecting laughter in spontaneous speech by constructing laughter bouts

Laughter frequently occurs in spontaneous speech (e.g. conversational speech, meeting speech). Detecting laughter is quite important for semantic analysis, highlight extraction, spontaneous speech recognition, etc. In this paper, we first analyze the ...

Comments

Information & Contributors

Information

Published In

SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval

August 2006

768 pages

ISBN:1595933697

DOI:10.1145/1148170

General Chair:
Efthimis N. Efthimiadis
University of Washington
,
Program Chairs:
Susan Dumais
Microsoft Research, Redmond
,
David Hawking
CSIRO ICT Centre, Canberra, Australia
,
Kalervo Järvelin,
University of Tampere, Finland

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 August 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR06

Sponsor:

SIGIR06: The 29th Annual International SIGIR Conference

August 6 - 11, 2006

Washington, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
375
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Skorkovská L(2016)Relevant Documents Selection for Blind Relevance Feedback in Speech Information RetrievalText, Speech, and Dialogue10.1007/978-3-319-45510-5_48(418-425)Online publication date: 3-Sep-2016
https://doi.org/10.1007/978-3-319-45510-5_48
Skorkovská L(2016)Comparison of Retrieval Approaches and Blind Relevance Feedback Methods Within the Czech Speech Information RetrievalSpeech and Computer10.1007/978-3-319-43958-7_21(182-190)Online publication date: 13-Aug-2016
https://doi.org/10.1007/978-3-319-43958-7_21
Ward NWerner SGarcia FSanchis E(2015)A prosody-based vector-space model of dialog activity for information retrievalSpeech Communication10.1016/j.specom.2015.01.00468:C(85-96)Online publication date: 1-Apr-2015
https://dl.acm.org/doi/10.1016/j.specom.2015.01.004
Skorkovská L(2015)Score Normalization Methods for Relevant Documents Selection for Blind Relevance Feedback in Speech Information RetrievalProceedings of the 18th International Conference on Text, Speech, and Dialogue - Volume 930210.1007/978-3-319-24033-6_36(316-324)Online publication date: 14-Sep-2015
https://dl.acm.org/doi/10.1007/978-3-319-24033-6_36
Galuščáková PPecina PKankanhalli MRueger SManmatha RJose Jvan Rijsbergen K(2014)Experiments with Segmentation Strategies for Passage Retrieval in Audio-Visual DocumentsProceedings of International Conference on Multimedia Retrieval10.1145/2578726.2578753(217-224)Online publication date: 1-Apr-2014
https://dl.acm.org/doi/10.1145/2578726.2578753
Skorkovská L(2014)First Experiments with Relevant Documents Selection for Blind Relevance Feedback in Spoken Document RetrievalSpeech and Computer10.1007/978-3-319-11581-8_29(235-242)Online publication date: 2014
https://doi.org/10.1007/978-3-319-11581-8_29
Turunen VKurimo MKeronen S(2013)Results for Variable Speaker and Recording Conditions on Spoken IR in FinnishProceedings of the 15th International Conference on Speech and Computer - Volume 811310.1007/978-3-319-01931-4_36(271-277)Online publication date: 1-Sep-2013
https://dl.acm.org/doi/10.1007/978-3-319-01931-4_36
Wartena C(2012)Comparing segmentation strategies for efficient video passage retrieval2012 10th International Workshop on Content-Based Multimedia Indexing (CBMI)10.1109/CBMI.2012.6269850(1-6)Online publication date: Jun-2012
https://doi.org/10.1109/CBMI.2012.6269850
Galuščáková PPecina PHajič J(2012)Penalty functions for evaluation measures of unsegmented speech retrievalProceedings of the Third international conference on Information Access Evaluation: multilinguality, multimodality, and visual analytics10.1007/978-3-642-33247-0_12(100-111)Online publication date: 17-Sep-2012
https://dl.acm.org/doi/10.1007/978-3-642-33247-0_12
Eskevich MMagdy WJones G(2012)New metrics for meaningful evaluation of informally structured speech retrievalProceedings of the 34th European conference on Advances in Information Retrieval10.1007/978-3-642-28997-2_15(170-181)Online publication date: 1-Apr-2012
https://dl.acm.org/doi/10.1007/978-3-642-28997-2_15
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Combining LVCSR and vocabulary-independent ranked utterance retrieval for robust speech search

Characterizing and detecting spontaneous speech: Application to speaker role recognition

Detecting laughter in spontaneous speech by constructing laughter bouts

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations