skip to main content
10.1145/3331184.3331421acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
abstract

Measuring Job Search Effectiveness

Published: 18 July 2019 Publication History

Abstract

Users of online job search websites interact with ranked lists of job summaries generated in response to queries, hoping to identify one or more jobs of interest. Hence, the quality of job search rankings becomes a primary factor that affects user satisfaction. In this work, we propose methodologies and measures for evaluating the quality of job search rankings from a user modeling perspective. We start by investigating job seekers' behavior when they are interacting with the generated rankings, leveraging job search interaction logs from Seek.com, a well-known Australasian job search website. The output of this investigation will be an accurate model of job seekers that will be incorporated into an effectiveness metric. Recent proposals for job search ranking models used using two types of metrics to evaluate the quality of the ranking generated by the models: (1) offline metrics, such as NDCG@k (k is set to the number of job summaries shown in the first page), Prec@1, or Mean Reciprocal Rank (MRR); and (2) online metrics, such as click-through rate and job application rate [3, 6].

References

[1]
L. Azzopardi, P. Thomas, and N. Craswell. Measuring the utility of search engine result pages. In Proc. SIGIR, pages 605--614, 2018.
[2]
B. Carterette, E. Kanoulas, and E. Yilmaz. Incorporating variability in user behavior into systems based evaluation. In Proc. CIKM, pages 135--144, 2012.
[3]
J. Li, D. Arya, V. Ha-Thuc, and S. Sinha. How to get them a dream job? Entity-aware features for personalized job search ranking. In Proc. KDD, pages 501--510, 2016.
[4]
A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Sys., 27(1):2.1--2.27, 2008.
[5]
A. Moffat, P. Bailey, F. Scholer, and P. Thomas. Incorporating user expectations and behavior into the measurement of search effectiveness. ACM Trans. Inf. Sys., 35(3):24:1--24:38, 2017.
[6]
A. Saha and D. Arya. Generalized mixed effect models for personalizing job search. In Proc. SIGIR, pages 1129--1132, 2017.
[7]
R. W. White and S. M. Drucker. Investigating behavioral variability in web search. In Proc. WWW, pages 21--30, 2007.
[8]
A. F. Wicaksono and A. Moffat. Empirical evidence for search effectiveness models. In Proc. CIKM, pages 1571--1574, 2018.

Cited By

View all
  • (2022)(Under)valuing lived experience in the disability workforce: A snapshot of Australian job recruitmentAustralian Journal of Social Issues10.1002/ajs4.23858:2(425-440)Online publication date: 22-Sep-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
July 2019
1512 pages
ISBN:9781450361729
DOI:10.1145/3331184
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2019

Check for updates

Author Tags

  1. evaluation
  2. job search
  3. user model

Qualifiers

  • Abstract

Funding Sources

  • SIGIR Travel Grant
  • SEEK.COM
  • The Australian Research Council

Conference

SIGIR '19
Sponsor:

Acceptance Rates

SIGIR'19 Paper Acceptance Rate 84 of 426 submissions, 20%;
Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)14
  • Downloads (Last 6 weeks)5
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)(Under)valuing lived experience in the disability workforce: A snapshot of Australian job recruitmentAustralian Journal of Social Issues10.1002/ajs4.23858:2(425-440)Online publication date: 22-Sep-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media