Article

Building an information retrieval test collection for spontaneous conversational speech

Authors:

Douglas W. Oard,

Dagobert Soergel,

David Doermann,

G. Craig Murray,

Jianqiang Wang,

Bhuvana Ramabhadran,

Samuel Gustman,

James Mayfield,

Liliya Kharevych,

Stephanie StrasselAuthors Info & Claims

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 41 - 48

https://doi.org/10.1145/1008992.1009002

Published: 25 July 2004 Publication History

Abstract

Test collections model use cases in ways that facilitate evaluation of information retrieval systems. This paper describes the use of search-guided relevance assessment to create a test collection for retrieval of spontaneous conversational speech. Approximately 10,000 thematically coherent segments were manually identified in 625 hours of oral history interviews with 246 individuals. Automatic speech recognition results, manually prepared summaries, controlled vocabulary indexing, and name authority control are available for every segment. Those features were leveraged by a team of four relevance assessors to identify topically relevant segments for 28 topics developed from actual user requests. Search-guided assessment yielded sufficient inter-annotator agreement to support formative evaluation during system development. Baseline results for ranked retrieval are presented to illustrate use of the collection.

References

[1]

EU-US working group on spoken-word audio collections, 2003. http://www.dcs.shef.ac.uk/spandh/projects/swag/.

[2]

C. Buckley and E. Voorhees. Evaluating evaluation measure stability. In SIGIR 2000, pages 33--40, 2000.

Digital Library

[3]

William Byrne et al. Automated recognition of spontaneous speech for access to multilingual oral history archives. IEEE Transactions on Speech and Audio Processing, 12(4), 2004.

[4]

J. Carletta. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249--254, 1996.

Digital Library

[5]

S. F. Chen and J. Goodman. An empirical study of smoothing techniques for language modeling. In Computer Speech and Language, 1999.

Digital Library

[6]

C. Cieri et al. Corpora for topic detection and tracking. In Topic Detection and Tracking: Event-Based Information Organization. Kluwer Academic, Boston, 2002.

Digital Library

[7]

C. Cleverdon. The Cranfield tests on index language devices. ASLIB Proceedings, 19(6):173--194, 1967.

[8]

G. V. Cormack et al. Efficient construction of large test collections. In SIGIR '98, pages 282--289, 1998.

Digital Library

[9]

Martin Franz et al. Ad hoc and multilingual information retrieval at IBM. In TREC-7, 1998.

[10]

J. S. Garofolo et al. The TREC spoken document retrieval track: A success story. In TREC-8, 1999.

[11]

J. Godfrey et al. SWITCHBOARD: telephone speech corpus for research and development. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 517--520, 1992.

[12]

Samuel Gustman et al. Supporting access to large digital oral history archives. In The Second Joint Conference on Digital Libraries, pages 18--27, 2002.

Digital Library

[13]

Xiaoli Huang and Dagobert Soergel. Relevance judges' understanding of topical relevance types: An explication of an enriched concept of topical relevance. In Annual Meeting of the American Society for Information Science and Technology, 2004. to appear.

[14]

R. V. Katter. The influence of scale form on relevance judgment. Information Storage and Retrieval, 4(1):1--11, 1968.

[15]

B. Ramabhadran et al. Towards automatic transcription of large spoken archives - English ASR for the MALACH project. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2003.

[16]

S. E. Robertson et al. Okapi at TREC-3. In TREC-3, pages 109--126, 1994.

[17]

T. Rong et al. Towards the identification of the optimal number of relevance categories. Journal of the American Society for Information Science, 50(3):254--264, 1998.

Digital Library

[18]

L. Schamber. Variations in relevance and information behavior. In Annual Review of Information Science and Technology, volume 29, pages 3--48. 2000.

[19]

K. Sparck-Jones and C. J. van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32(1):59--72, 1976.

[20]

E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management, 36:697--716, 2000.

Digital Library

[21]

S. Whittaker et al. Play it again: A study of the factors underlying search browsing behavior. In CHI '98, pages 247--248, 1998.

Digital Library

[22]

Y. Yang and X. Liu. A re-examination of text categorization methods. In SIGIR '99, 1999.

Digital Library

Cited By

Hu XLee JBainbridge DChoi KOrganisciak PDownie J(2017)The MIREX grand challengeJournal of the Association for Information Science and Technology10.1002/asi.2361868:1(97-112)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1002/asi.23618
McNeese NReddy M(2017)The role of team cognition in collaborative information seekingJournal of the Association for Information Science and Technology10.1002/asi.2361468:1(129-140)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1002/asi.23614
Thelwall MGoriunova OVis FFaulkner SBurns AAulich JMas-Bleda AStuart ED'Orazio F(2016)Chatting through pictures? A classification of images tweeted in one week in the UK and USAJournal of the Association for Information Science and Technology10.1002/asi.2362067:11(2575-2586)Online publication date: 1-Nov-2016
https://dl.acm.org/doi/10.1002/asi.23620
Show More Cited By

Index Terms

Building an information retrieval test collection for spontaneous conversational speech
1. Information systems
  1. Information retrieval

Recommendations

Characterizing and detecting spontaneous speech: Application to speaker role recognition

Processing spontaneous speech is one of the many challenges that automatic speech recognition systems have to deal with. The main characteristics of this kind of speech are disfluencies (filled pause, repetition, false start, etc.) and many studies have ...
Investigating cross-language speech retrieval for a spontaneous conversational speech collection
NAACL-Short '06: Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers

Cross-language retrieval of spontaneous speech combines the challenges of working with noisy automated transcription and language translation. The CLEF 2005 Cross-Language Speech Retrieval (CL-SR) task provides a standard test collection to investigate ...
Automatic Detection of Speech Disfluencies in the Spontaneous Russian Speech
SPECOM 2013: Proceedings of the 15th International Conference on Speech and Computer - Volume 8113

Spontaneous speech is rarely fluent due to human nature. And among other characteristics of spontaneous speech there are the speech variation and the presence of speech disfluencies such as hesitations, fillers, artefacts. Such elements are an obstacle ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

July 2004

624 pages

ISBN:1581138814

DOI:10.1145/1008992

General Chair:
Mark Sanderson
University of Sheffield (UK)
,
Program Chairs:
Kalervo Järvelin
University of Tampere (Finland)
,
James Allan
University of Massachusetts (USA)
,
Peter Bruza
Distributed Systems Technology Centre (Australia)

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR04

Sponsor:

SIGIR04: The 27th ACM/SIGIR International Symposium on Information Retrieval 2004

July 25 - 29, 2004

Sheffield, United Kingdom

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
922
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hu XLee JBainbridge DChoi KOrganisciak PDownie J(2017)The MIREX grand challengeJournal of the Association for Information Science and Technology10.1002/asi.2361868:1(97-112)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1002/asi.23618
McNeese NReddy M(2017)The role of team cognition in collaborative information seekingJournal of the Association for Information Science and Technology10.1002/asi.2361468:1(129-140)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1002/asi.23614
Thelwall MGoriunova OVis FFaulkner SBurns AAulich JMas-Bleda AStuart ED'Orazio F(2016)Chatting through pictures? A classification of images tweeted in one week in the UK and USAJournal of the Association for Information Science and Technology10.1002/asi.2362067:11(2575-2586)Online publication date: 1-Nov-2016
https://dl.acm.org/doi/10.1002/asi.23620
Golub KSoergel DBuchanan GTudhope DLykke MHiom D(2016)A framework for evaluating automatic indexing or classification in the context of retrievalJournal of the Association for Information Science and Technology10.1002/asi.2360067:1(3-16)Online publication date: 1-Jan-2016
https://dl.acm.org/doi/10.1002/asi.23600
Cui JMamou JKingsbury BRamabhadran B(2014)Automatic keyword selection for keyword search development and tuning2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2014.6855126(7839-7843)Online publication date: May-2014
https://doi.org/10.1109/ICASSP.2014.6855126
Wang JOard D(2012)Matching meaning for cross-language information retrievalInformation Processing and Management: an International Journal10.1016/j.ipm.2011.09.00348:4(631-653)Online publication date: 1-Jul-2012
https://dl.acm.org/doi/10.1016/j.ipm.2011.09.003
Senay GLinares GLecouteux B(2011)A segment-level confidence measure for Spoken Document Retrieval2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2011.5947616(5548-5551)Online publication date: May-2011
https://doi.org/10.1109/ICASSP.2011.5947616
Chelba CHazen TRamabhadran BSaraçlar M(2011)Speech RetrievalSpoken Language Understanding10.1002/9781119992691.ch15(417-446)Online publication date: 23-Mar-2011
https://doi.org/10.1002/9781119992691.ch15
Akbacak MHansen J(2010)Spoken Proper Name Retrieval for Limited Resource Languages Using Multilingual Hybrid RepresentationsIEEE Transactions on Audio, Speech, and Language Processing10.1109/TASL.2009.203578518:6(1486-1495)Online publication date: 1-Aug-2010
https://dl.acm.org/doi/10.1109/TASL.2009.2035785
Alzghool MInkpen DLarson MOrdelman Rde Jong FKohler JKraaij W(2009)Exploring fusion in a spontaneous speech retrieval taskProceedings of the third workshop on Searching spontaneous conversational speech10.1145/1631127.1631130(11-20)Online publication date: 23-Oct-2009
https://dl.acm.org/doi/10.1145/1631127.1631130
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten