needle in the haystack
Crowd Evaluations: The needle in the haystack
How crowds can help to find the best idea (08.09.2022)
Förderjahr 2020 / Stipendien Call #15 / ProjektID: 5023 / Projekt: Sabotage in crowdsourcing

Crowdsourcing contests can lead to enormous amounts of contributions. This article discusses how the crowd can help to solve this problem.

During the devastating Deepwater Horizon accident, British Petroleum (BP) reached outside company boundaries for suggestions from external contributors. More specifically, they started a crowdsourcing contest and asked the general public what could be done to deal with the catastrophe. BP hoped that external contributors would find creative solutions to the pressing problem of the oil spill.

The result of their approach was astonishing: A crowd of external contributors from a vast array of knowledge domains responded to the open call and provided over 120,000 suggestions. Therefore, BP received one of the largest pools of external suggestions ever documented.

However, the success of the idea crowdsourcing resulted in a new problem: How and by whom should the submitted ideas be evaluated?

The evaluation of such large amounts of contributions requires a lot of resources if traditional expert panels are used for this purpose (Criscuolo et al., 2017). For example, IBM employed 50 senior executives for several weeks in order to evaluate over 50,000 ideas that were submitted during their “Innovation Jams” (Bjelland & Wood, 2008). Similarly, it took Google 3,000 employees and “much longer than expected” to evaluate over 150,000 submissions of their 10^100 project (Google, 2009). Estimates suggest that it takes about 500$ and four hours to evaluate one idea in a Fortune 100 company (Klein & Garcia, 2015). In the case of Google this estimation would result in costs of over 60,000,000$ for the evaluation of the contributions alone.

One possible solution to the problem of finding the best idea is the crowd itself. Outsourcing the screening process to the crowd is usually referred to as crowd evaluation or crowd voting. Current research findings indicate that crowds can indeed compete with experts when evaluating ideas. For example, Magnussen et al. (2016) show “that companies can employ users during the initial screening process using criteria assessment to select the best ideas for further elaboration, something that would significantly reduce the number of ideas”.

If applied properly, crowd evaluations seem to be a promising method for efficient evaluation tasks like idea screening. However, crowd evaluations are not just used for idea assessment. A much more known form of crowd evaluations are consumer reviews on the internet – a topic, which I will touch upon in my next blog post.



Bjelland, O. M., & Wood, R. C. (2008). An inside view of IBM’s’ Innovation Jam’. MIT Sloan Management Review, 50(1), 32.

Criscuolo, P., Dahlander, L., Grohsjean, T., & Salter, A. (2017). Evaluating Novelty: The Role of Panels in the Selection of R&D Projects. Academy of Management Journal, 60(2), 433–460.

Google. (2009, September 24). Announcing Project 10^100 idea themes. Official Google Blog.…

Klein, M., & Garcia, A. C. B. (2015). High-speed idea filtering with the bag of lemons. Decision Support Systems, 78, 39–50.

Magnusson, P. R., Wästlund, E., & Netz, J. (2016). Exploring Users’ Appropriateness as a Proxy for Experts When Screening New Product/Service Ideas: Exploring Users as a Proxy for Expert Judges. Journal of Product Innovation Management, 33(1), 4–18.


Crowdsourcing Wisdom of the Crowds Innovation User Innovation Innovationswettbewerb

Richard Olbrecht

Profile picture for user RichardOlbrecht
@ the Institute for Entrepreneurship & Innovation, WU Vienna University of Economics and Business
Diese Frage dient der Überprüfung, ob Sie ein menschlicher Besucher sind und um automatisierten SPAM zu verhindern.
    Der datenschutzrechtliche Verantwortliche (Internet Privatstiftung Austria - Internet Foundation Austria, Österreich) würde gerne mit folgenden Diensten Ihre personenbezogenen Daten verarbeiten. Dies ist für die Nutzung der Website nicht notwendig, ermöglicht aber eine noch engere Interaktion mit Ihnen. Falls gewünscht, treffen Sie bitte eine Auswahl: