short-paper

Public Access

A Test Collection for Relevance and Sensitivity

Authors:

Mahmoud F. Sayed,

William Cox,

Jonah Lynn Rivera,

Caitlin Christian-Lamb,

Modassir Iqbal,

Douglas W. Oard,

Katie ShiltonAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1605 - 1608

https://doi.org/10.1145/3397271.3401284

Published: 25 July 2020 Publication History

PDF eReader

Abstract

Recent interest in the design of information retrieval systems that can balance an ability to find relevant content with an ability to protect sensitive content creates a need for test collections that are annotated for both relevance and sensitivity. This paper describes the development of such a test collection that is based on the Avocado Research Email Collection. Four people created search topics as a basis for assessing relevance, and two personas describing the sensitivities of representative (but fictional) content creators were created as a basis for assessing sensitivity. These personas were based on interviews with potential donors of historically significant email collections and with archivists who currently manage access to such collections. Two annotators then created relevance and sensitivity judgments for 65 topics, divided approximately equally between the two personas. Annotator agreement statistics indicate fairly good external reliability for both relevance and sensitivity annotations, and a baseline sensitivity classifier trained and evaluated using cross-validation achieved better than 80% $F_1$, suggesting that the resulting collection will likely be useful as a basis for comparing alternative retrieval systems that seek to balance relevance and sensitivity.

Supplementary Material

MP4 File (3397271.3401284.mp4)

Download
12.09 MB

References

[1]

R Artstein and M Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34, 4 (2008), 555--596.

Digital Library

Google Scholar

[2]

A Cooper. 2004. The origin of personas. Innovation 23, 1 (2004), 26--29.

Google Scholar

[3]

G Cormack et al. 2010. Overview of the TREC 2010 Legal Track. In TREC.

Google Scholar

[4]

M Hearst. 2005. Teaching applied NLP: Triumphs and tribulations. In ACL Workshop on Effective Tools and Methodologies for Teaching NLP and CL.

Google Scholar

[5]

S Jabbari et al. 2006. Towards the Orwellian Nightmare: Separation of Business and Personal Emails. In COLING/ACL poster sessions.

Google Scholar

[6]

K Martin and H Nissenbaum. 2016. Measuring Privacy: An Empirical Test Using Context to Expose Confounding Variables. Colum. Sci. & Tech. L. Rev.18 (2016).

Google Scholar

[7]

G McDonald et al. 2017. Enhancing Sensitivity Classification with Semantic Features Using Word Embeddings. In ECIR.

Google Scholar

[8]

D Mulligan et al. 2016. Privacy is an essentially contested concept: a multi-dimensional analytic for mapping privacy. Phil. Trans. Royal Soc. A374 (2016).

Google Scholar

[9]

H Nissenbaum. 2009. Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press.

Digital Library

Google Scholar

[10]

D Oard et al. 2015. Avocado Research Email Collection. LDC2015T03.

Google Scholar

[11]

J Pruitt and T Adlin. 2010.The persona lifecycle: keeping people in mind throughout product design. Elsevier.

Google Scholar

[12]

M Sayed et al.2019. Jointly Modeling Relevance and Sensitivity for Search Among Sensitive Content. In SIGIR.

Google Scholar

Cited By

View all

McKechnie J(2024)Cascading Ranking Pipelines for Sensitivity-Aware SearchAdvances in Information Retrieval10.1007/978-3-031-56069-9_41(331-333)Online publication date: 23-Mar-2024
https://doi.org/10.1007/978-3-031-56069-9_41
Shah HJaidka KUngar LFagan JGrosser T(2023)Building a Multimodal Classifier of Email Behavior: Towards a Social Network Understanding of Organizational CommunicationInformation10.3390/info1412066114:12(661)Online publication date: 14-Dec-2023
https://doi.org/10.3390/info14120661
Baron JSayed MOard D(2022)Providing More Efficient Access to Government Records: A Use Case Involving Application of Machine Learning to Improve FOIA Review for the Deliberative Process PrivilegeJournal on Computing and Cultural Heritage 10.1145/348104515:1(1-19)Online publication date: 22-Jan-2022
https://dl.acm.org/doi/10.1145/3481045
Show More Cited By

Index Terms

A Test Collection for Relevance and Sensitivity
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Test collections

Recommendations

Video Test Collection with Graded Relevance Assessments
CHIIR '16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval

Relevance is a complex, but core, concept within the field of Information Retrieval. In order to allow system comparisons the many factors that influence relevance are often discarded to allow abstraction to a single score relating to relevance. This ...
A Test Collection for Ad-hoc Dataset Retrieval
SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

This paper introduces a new test collection for ad-hoc dataset retrieval, which have been developed through a shared task called Data Search in the fifteenth NTCIR. This test collection consists of dataset collections derived from the US and Japanese ...
Deriving a test collection for clinical information retrieval from systematic reviews
DTMBIO '10: Proceedings of the ACM fourth international workshop on Data and text mining in biomedical informatics

In this paper, we describe the construction of a test collection for evaluating clinical information retrieval. The purpose of this test collection is to provide a basis for researchers to experiment with PECO-structured queries. Systematic reviews are ...

Comments

Information & Contributors

Information

Published In

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Division of Information and Intelligent Systems

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
421
Total Downloads

Downloads (Last 12 months)146
Downloads (Last 6 weeks)25

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

McKechnie J(2024)Cascading Ranking Pipelines for Sensitivity-Aware SearchAdvances in Information Retrieval10.1007/978-3-031-56069-9_41(331-333)Online publication date: 23-Mar-2024
https://doi.org/10.1007/978-3-031-56069-9_41
Shah HJaidka KUngar LFagan JGrosser T(2023)Building a Multimodal Classifier of Email Behavior: Towards a Social Network Understanding of Organizational CommunicationInformation10.3390/info1412066114:12(661)Online publication date: 14-Dec-2023
https://doi.org/10.3390/info14120661
Baron JSayed MOard D(2022)Providing More Efficient Access to Government Records: A Use Case Involving Application of Machine Learning to Improve FOIA Review for the Deliberative Process PrivilegeJournal on Computing and Cultural Heritage 10.1145/348104515:1(1-19)Online publication date: 22-Jan-2022
https://dl.acm.org/doi/10.1145/3481045
Sayed MMallekav NOard D(2022)Comparing Intrinsic and Extrinsic Evaluation of Sensitivity ClassificationAdvances in Information Retrieval10.1007/978-3-030-99739-7_25(215-222)Online publication date: 5-Apr-2022
https://doi.org/10.1007/978-3-030-99739-7_25
Iqbal MShilton KSayed MOard DRivera JCox W(2021)Search with DiscretionProceedings of the ACM on Human-Computer Interaction10.1145/34492075:CSCW1(1-20)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3449207

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Video Test Collection with Graded Relevance Assessments

A Test Collection for Ad-hoc Dataset Retrieval

Deriving a test collection for clinical information retrieval from systematic reviews