skip to main content
10.1145/3583780.3615506acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
abstract

Application and Evaluation of Large Language Models for the Generation of Survey Questions

Published: 21 October 2023 Publication History

Abstract

Generative Language Models have shown promising results in various domains, and some of the most successful applications are related to "concept expansion", which is the task of generating extensive text based on concise instructions provided through a "seed" prompt. In this presentation we will discuss the recent work conducted by the Data Science team at SurveyMonkey, where we have recently introduced a new feature that harnesses Generative AI models to streamline the survey design process. With this feature users can effortlessly initiate this process by specifying their desired objectives through a prompt, allowing them to automate the creation of surveys that include the critical aspects they wish to investigate.
We will share our findings regarding some of the challenges encountered during the development of this feature. These include techniques for conditioning the model outputs, integrating generated text with industry-standard questions, fine-tuning Language Models using semi-synthetic Data Generation techniques, and more. Moreover, we will showcase the Evaluation Methodology that we have developed to measure the quality of the generated surveys across several dimensions. This evaluation process is crucial in ensuring that the generated surveys align well with user expectations and serve their intended purpose effectively. Our goal is to demonstrate the promising potential of Generative Language Models in the context of Survey Research, and we believe that sharing our learnings on these challenges and how we addressed them will be useful for practitioners working with Language Models on similar problems.

Cited By

View all
  • (2024)“If the End Justifies the Means, Then This Is Permissible…”: Academic Deviations of Russian Postgraduate Students in Socio-Humanitarian AreasVysshee Obrazovanie v Rossii = Higher Education in Russia10.31992/0869-3617-2024-33-3-84-10333:3(84-103)Online publication date: 5-Apr-2024
  • (2024)ChatGPTest: Opportunities and Cautionary Tales of Utilizing AI for Questionnaire PretestingField Methods10.1177/1525822X241280574Online publication date: 12-Sep-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management
October 2023
5508 pages
ISBN:9798400701245
DOI:10.1145/3583780
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2023

Check for updates

Author Tags

  1. generative AI
  2. survey research
  3. text evaluation

Qualifiers

  • Abstract

Conference

CIKM '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)111
  • Downloads (Last 6 weeks)6
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)“If the End Justifies the Means, Then This Is Permissible…”: Academic Deviations of Russian Postgraduate Students in Socio-Humanitarian AreasVysshee Obrazovanie v Rossii = Higher Education in Russia10.31992/0869-3617-2024-33-3-84-10333:3(84-103)Online publication date: 5-Apr-2024
  • (2024)ChatGPTest: Opportunities and Cautionary Tales of Utilizing AI for Questionnaire PretestingField Methods10.1177/1525822X241280574Online publication date: 12-Sep-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media