Putting the crowd to work in a knowledge-based factory
Introduction
We present Crowdsourcing as a tool to facilitate machine intelligence in a knowledge-based factory. Our approach acknowledges the outstanding capacity of the human brain, but rather than trying to understand, or mimic, the complexity of the cognitive processes, we propose that human intelligence is employed directly at critical steps where machine intelligence cannot match human performance.
A direct consequence of this methodology is that rather than systems requiring, say, rule-bases, inference-engines, fitness functions or databases of “case” examples, the focus becomes the construction of queries for the crowd and the development of statistical methods for the aggregation of their responses. However although this scenario (i.e. where Crowdsourcing provides the reasoning functions traditionally carried out by AI subsystems) will change the nature of individual software components, the overall architecture of problem solving systems will, in many cases, be unchanged. But Crowdsourcing offers the opportunity to do more than simply provide a neat on-line interface to human reasoning and judgement. It offers the opportunity to discover effective problem solving strategies. By definition, “machine intelligence” systems are created by programmers who have encoded problem solving strategies in the software. So while occasionally emergent behaviours are observed, we would argue that for the most part AI systems solve problems exactly has they have been design too. And this is both their limitation and their strength.
Crowdsourcing, in contrast, solves problems, cleans up data, classifies content, selects options, creates new content, and many other tasks using strategies that appear ‘opaque’ to the user. But because of the digital nature of the activity, there is an opportunity to record, observe and assess the problem solving strategies of many individuals in a way that would be extremely difficult to do in any other scenario. To illustrate this, Section 3 of the paper presents the early results of Crowdsourced part nesting where not only do the results improve upon those generated by commercial CAM systems but could, potentially, also provide insights into how automate system could be improved.
Finally, crucial to any industrial use of Crowdsourced “human intelligence”, is the constant availability of sufficient quantity, and quality, of on-line workers. Regardless of the task undertaken, our results suggest that the Internet is now sufficiently large and globally distributed well that commercial Crowdsourcing sites can easily provide results in only a few hours on a 24/7 basis.
This paper is divided into five sections: this introduction continues with a brief overview of machine reasoning in the knowledge-based factory and describing the emerging technology of Internet Crowdsourcing. The next section (Section 2) describes how Crowdsourcing has been use to provide reasoning for a 3D content based retrieval system whose overall architecture is analogous to “traditional” AI applications for machine vision or speech recognition. The following section (Section 3) illustrates the opportunity for Crowdsourcing to contribute insights to problem solving strategies (in this case 2D shape nesting) as well as results. Finally, a discussion (Section 4) summarises the potential applications and the challenges of Crowdsourcing industrial tasks, before conclusions are drawn, in Section 5.
Machine intelligence has been used to support industrial processes that range from computer vision for robotics to creative design. Duffy [1] presents six main machine learning techniques: agent-based learning, analogical reasoning, induction methods, genetic algorithms, knowledge compilation, and neural networks. A common element of all these methods is the need to “inform” or “teach” the system using databases of examples.
For example, analogical reasoning (i.e. finding solutions to problems based on retrieving knowledge from previous experiences), induction methods (i.e. where knowledge is generated by the amalgamation of similar data and its analysis to obtain a classification), genetic algorithms (i.e. when new concepts are generated by the cross-over or ‘mutation’ of previous ones), knowledge compilation (i.e. simplify into more fundamental knowledge, so it can be reusable in other situations), and neural networks (when a machine, executes a similar learning mechanism to a human brain by training on example data) are all strategies that could also be employed in conjunction with Crowdsourced databases of examples to create a true ‘knowledge-based’ factory.
Currently, when embedded in an application, AI technologies (such as those listed above) are rarely able to work on raw data (e.g. documents, audio or image files etc.). More typically the data is analysed to identify features, or characteristics, which form the ‘language’ of the reasoning system (Fig. 1).
In a machine vision system, the features might be “edges” identifies in a .jpeg image, in speech recognition software signal processing is used to identify phonetic patterns and in the 3D CAM depressions are identified on geometric CAD models prior to process planning. This model is unchanged by Crowdsourcing technology and the following sections demonstrates how micro-outsourcing to the Internet can provide the functionality for both the ‘Feature Recognition’ and ‘Reasoning’ stages that are characteristic of so many problem solving architectures.
The term ‘crowdsourcing’ was coined by Jeff Howe in 2006 as “the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call” [2]. These activities are executed by people who do not necessarily know each other, and interact with the company, the ‘requester’, via virtual tools and an internet connection. They become ‘the workers’: they can access tasks, execute them, upload the results and receive various forms of payment using any web browser. This is a labour market open 24/7, with a diverse workforce available to perform tasks quickly and cheaply.
The crowdsourcing platform used in the investigations reported here was Amazon’s mTurk (www.mturk.com) which was selected because of the large number of worker available. Although it should be note that there are several alternatives (e.g. HumanGrid: http://www.humangrid.de and Crowdflower: crowdflower.com).
As shown in Fig. 2, the ‘requesters’ both design and post tasks for the Crowd to work on. In mTurk, tasks given to the ‘workers’ are called ‘HITs’ (Human Intelligence Tasks). Requesters can test workers before allowing them to accept tasks and so establish a baseline performance level of prospective workers. Requesters can also accept, or reject, the results submitted by the workers, and this decision impacts on the worker’s reputation within the mTurk system. Payments for completed tasks can be redeemed as ‘Amazon.com’ gift certificates or alternatively transferred to a worker’s bank account. Details of the mTurk interface design, how an API is used to create and post HITs and a description of the workers’ characteristics are beyond the scope of this paper but can be found (along with further details of the experimental results) in [3], [4]. With each result submitted by a worker the requester receives an answer that including various information about how the task was processed. One element of this data is an unique “workerID” allowing the requester to distinguish between individual workers. Using this “workerID” it is possible to analyse how many different HITs each worker completed.
A definitive classification of Crowdsourcing tasks has not yet been established, however Corney et al. [5] suggest three possible categorisations based upon: nature of the task (creation, evaluation and organisation tasks), nature of the crowd (‘expert’, ‘most people’ and ‘vast majority’) and nature of the payment (voluntary contribution, rewarded at a flat rate and rewarded with a prize). Similarly Crowdsourcing practitioners, such as Chaordix (from the Cambrian House [6]) describes Crowdsourcing models as a Contest (i.e. individual submit ideas and the winner is selected by the company, ‘the requester’), a Collaboration (i.e. individuals submit their ideas or results, the crowd evolves the ideas and picks a winner), and Moderated (i.e. individuals submit their ideas, the crowd evolves those ideas, a panel – set by ‘the requesters’ select the finalists and the crowd votes on a winner). In the last few years academics across many different disciplines have started reporting the use of Internet Crowdsourcing to support a range of research projects, e.g. social network motivators [7], relevance of evaluations and queries [8], [9], accuracy in judgement and evaluations[10]. Despite this activity few industrial applications of Crowdsourcing have been reported and this gap in the literature motivated the authors to undertake the studies into 3D search and 2D part nesting reported in the following sections.
Section snippets
3D Search case study
At present, 3D models (e.g. engineering drawings) are indexed by alpha numeric ‘part-numbers’ with a format unique to each individual organisation. Although this indexing system works well in the context of on-going maintenance and development of individual parts, it offers little scope for ‘data-mining’ (i.e. exploration) of an organisation’s inventory of designs. In addition to the sourcing of parts the application of a 3D similarity matching algorithm to large collections of parts would
Crowdsourcing 2D part nesting
Other AI reasoning applications such a planning require very different approaches from those used in classification or recognition problems. Typically networks of constraints are constructed and the “intelligence” or problem solving strategy is embedded in the algorithm used to navigate this graph [19] Crowdsourcing offers the possibility of solving planning, and other combinatorially explosive, problems using distributed human labour. To investigate this possibility, we created an experiment
Discussion
In this paper, we have shown how Crowdsourced can be used to carry out a variety of geometric reasoning tasks. This is a powerful testament to the flexibility of the process presented here. Although the design and coding required to implement the HITs was not trivial, it was considerably easier than the development of new algorithms for machine cognition and learning. The crucial aspect was to formulate the right question and carefully consider the instructions provided to the mTurk workers.
In
Conclusions
Examples of crowdsourced work for various mechanical CAD/CAM applications have been presented in this paper. Beyond simply establishing that the approach produced surprisingly good results we learnt that it was important to present the “right question” to the crowd. Therefore, the sophisticated job was simplified into several steps to ensure clarity and comprehension by the workers. Interestingly, like Crowdsourcing applications in other domains we have shown that preparation work (e.g. best
References (25)
- et al.
A solid modeling library for the World Wide Web
Computer Networks and ISDN Systems.
(1998) - et al.
Automated design of sheet metal punches for bending multiple parts in a single setup
Robotics and Computer-Integrated Manufacturing
(2001) - et al.
Retrieving 3D CAD model by freehand sketches for design reuse
Advanced Engineering Informatics
(2008) - et al.
Developing an engineering shape benchmark for CAD models
Computer-Aided Design
(2006) - et al.
Models for production planning under uncertainty: a review
International Journal of Production Economics
(2006) - et al.
Models and algorithms for three-stage two-dimensional bin packing
European Journal of Operational Research
(2007) The “What” and “How” of learning in design
IEEE Expert: Intelligent Systems and Their Applications
(1997)- J. Howe, “The rise of Crowdsourcing: http://www.wired.com/wired/archive/14.06/crowds_pr.html”, Wired, issue 14,...
- P. Jagadeesan, J. Wenzel, et al., Geometric Reasoning via Internet CrowdSourcing 2009 SIAM/ACM Joint Conference on...
- P. Jagadeesan, J. Wenzel, et al., Validation of purdue engineering shape benchmark clusters by Crowdsourcing, in:...
Outsourcing labour to the cloud
International Journal of Innovation and Sustainable Development
Cited by (31)
General framework, opportunities and challenges for crowdsourcing techniques: A Comprehensive survey
2020, Journal of Systems and SoftwareCitation Excerpt :Workers may be recruited based on their initial evaluation before participating in the CS process and reputation after they have completed some tasks. The former applies to the workers’ profile in which skill levels and expertise are determined through pre-qualification tasks, gold questions or entry questions at the time of recruitment Corney et al. (2010). Whereas, the latter based on the validation of completed tasks Mashhadi and Capra (2011).
Harnessing user innovation for social media marketing: Case study of a crowdsourced hamburger
2018, International Journal of Information ManagementCrowdsourcing with online quantitative design analysis
2018, Advanced Engineering InformaticsCitation Excerpt :Crowd sourcing in design is thought to be particularly effective if conducted early in the design process through participatory design [12–15]. In recent years online web-based tools [16–19] are enabling residents and members of the public to submit their feedback. This enables many people to provide comment on design choices, contributing their insight by geospatial tagging [16,3] to highlight design issues.
Multi-faceted assessment of trademark similarity
2016, Expert Systems with ApplicationsAn evaluation methodology for crowdsourced design
2015, Advanced Engineering InformaticsCitation Excerpt :These are generous payment levels in comparison to other reported research studies which could as low as $0.01 [35], $0.10 [36,35]. However, in consideration of the experiment’s level of difficulty, the lowest rate was fixed as $0.15 [37,38]. By choosing $1.00 as a maximum payment the task would by one of the best paid on the platform where only some translation jobs might be paid as much as $1.40 per hour [35,39].