Evaluating mass knowledge acquisition using the ALICE chatterbot: The AZ-ALICE dialog system
Introduction
The process of knowledge acquisition is to transfer existing knowledge and its structure into a computer-interpretable form (Potter, 2001). This knowledge can come from humans or other sources such as textual documents or encyclopedias. When coupled with the Internet, knowledge acquisition inherits new problems of scale such as information quality and reliability issues.
This paper investigates the knowledge acquisition activities of a chatterbot program that mimics human conversation. Web-based chatterbot systems can provide an easy, natural extension to knowledge acquisition. This style of dialog system, due to its robustness, scalability, and ease of connecting to the web for information retrieval, appears to be a viable approach for knowledge acquisition. One of the better performers in the field is the ALICEbot. ALICE, or Artificial Linguistic Internet Chat Entity, was developed by Richard Wallace in 1995. This system has had marked success, winning the Loebner Prize for most human-like computer in 2000, 2001, and 2004.
In this paper, we will investigate the existing literature in Section 2, going from knowledge acquisition and its many approaches, down to the ALICE chatterbots and how they fit into the framework. In Section 3, we introduce a set of research questions and offer possible hypotheses. In Section 4, we explain the system design implemented in our study. Section 5 looks at the experimental design in detail. Section 6 describes the results from the experiment and offers a discussion of their meaning. Finally, in Section 7, the conclusions and future directions are provided.
Section snippets
Literature review
Knowledge acquisition has been a sought after goal since the early days of Artificial Intelligence. Newell posited that psychology and structure are important elements to perform a sequence of complex tasks, and noted the similarities between cognitive tasks and existing programming languages that are engineered to use logic and conditional operators (Newell, 1973) to mimic human ability and to simulate human behavior (Feigenbaum and Simon, 1962).
Under the broad umbrella of knowledge
Research questions
Dialog systems can function in one of two ways; they can provide brief, concise, or well-detailed answers to a particular query or they can engage the user in providing small talk types of conversational responses.
This leads to our exploration of mass knowledge acquisition where our aim is two-fold. First, we explore using human subjects to train various dialog systems and study the impact of the acquired knowledge. Second, we study the effects of domain answers to those of the conversational
System design
To answer the questions posed, we constructed the AZ-ALICE dialog system. AZ-ALICE is built upon the freely available java-based ALICE Program D500 from www.ALICEbot.org. Our system can be broken into five component parts; the Chat User Interface, Chat Engine, AIML (Artificial Intelligence Markup Language) knowledge files, a Logging component, and Evaluation module.
The Chat User Interface is an XML-based web page that allows users to authenticate themselves and chat with the system. The system
The experiment
In our experiment, we created two chatterbots, BaseBot and TeleComm. BaseBot, the general conversational chatterbot, was our control chatterbot. It was an off-the-shelf ALICE ProgramD chatterbot loaded with the ‘Standard AIML’ rule set consisting of 23,735 knowledge categories that can be freely obtained from www.alicebot.org. Each of the knowledge categories consists of a pattern to match against the user input and a template response corresponding to the pattern. The other chatterbot,
Number of corrections made
In our investigation of mass knowledge acquisition, users were instructed to make corrections to the knowledge sets of Study 1 that would then be incorporated into the knowledge sets used by Study 2. The following user/chatterbot interaction followed this pattern:
User in Study 1: What are you talking about?
AZ-ALICE: The topic is mood are you in.
User's new suggested response: My bad.
User in Study 2: What are you talking about?
AZ-ALICE: My bad.
From this acquisition interaction, users entered 1707
Conclusions and future directions
From our study we can conclude that the use of a chatterbot as a knowledge acquisition tool appears to be a stable instrument in gathering both conversation and domain-related knowledge. We believe that with the decrease in correction rates observed between studies, that after several rounds of such corrections, that the knowledge base will be of sufficient quality to answer domain-related questions. Extending this research in such a way demonstrates the viability of having users train a
Acknowledgments
This work was supported in part by the NSF, ITR: “COPLINK Center for Intelligence and Security Informatics Research” Sept. 1, 2003–Aug. 31, 2005.
References (23)
- et al.
Artificial paranoia
Artificial Intelligence
(1971) An evaluation of strategies for selectively verifying utterance meanings in spoken natural language dialog
International Journal of Human-Computer Studies
(1998)- Anonymous, 2003. Home Page of The Loebner Prize–“The First Turing Test”,...
- et al.
Decision-theoretic case-based reasoning
IEEE Transactions on Systems, Man and Cybernetics, Part A
(1996) - et al.
Simulation of human verbal learning behavior
Communications of the ACM
(1962) Incremental knowledge acquisition in supervised learning networks
IEEE Transactions on Systems, Man and Cybernetics, Part A
(1996)- et al.
Case-based knowledge and induction
IEEE Transactions on Systems, Man and Cybernetics, Part A
(2000) - Han, S., Kim, Y., 2001. Intelligent dialogue system for plane euclidean geometry learning. International Conference on...
- et al.
Cobot in LambdaMOO: A Social Statistics Agent
(2000) The Study of the Application of a Keywords-based Chatbot System on the Teaching of Foreign Languages
(2002)
Cited by (12)
A cultural heritage framework using a Deep Learning based Chatbot for supporting tourist journey
2021, Expert Systems with ApplicationsCitation Excerpt :Nevertheless, these conversational agents required many manual effort for rules definition (mainly based on the application domain) at the base of the model. In the last years question–answer sites and social media generated a large amount of data leading to arise data-driven approaches (Ritter, Cherry, & Dolan, 2011; Jafarpour, Burges, & Ritter, 2010) to provide more human-like responses with respect to the rule-based methods (Schumaker et al., 2006). These approaches can be divided into retrieval based and generator model approaches on the basis of how they generate the answers.
A deep learning based chatbot for cultural heritage
2020, Proceedings of the ACM Symposium on Applied ComputingReview of interactive communication systems for business-to-business (B2B) services
2018, IS and T International Symposium on Electronic Imaging Science and TechnologyContent-Oriented User Modeling for Personalized Response Ranking in Chatbots
2018, IEEE/ACM Transactions on Audio Speech and Language ProcessingIncorporating loose-structured knowledge into conversation modeling via recall-gate LSTM
2017, Proceedings of the International Joint Conference on Neural NetworksA conversational agent for an online mental health intervention
2016, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)