Skip to main content

Early Detection of Collective Misconceptions with Network-Aware Machine Learning Tools

NSF Award # 1755873  (March 15, 2018 – February 28, 2021)

This project builds theory, algorithms, and frameworks that can be used to design network-aware machine learning tools aimed at eliciting useful diversity and improving the accuracy of collective forecasting. Researchers in the social and economic sciences know that there is great capacity for collective intelligence to emerge from Web-based systems. Yet herding and homophily effects often restrain the wisdom of crowds, vastly limiting this potential. The research furthers the study of complex systems by introducing a new framework that improves our understanding of the mechanisms that govern decision-making under social influence. Advancing complex systems theory in this way greatly enhances the ability to predict when crowds will provide accurate decision-making support for complex problems and when they will fail miserably. Further, the research aids the development of opinion aggregation mechanisms that efficiently capitalize on diversity. The planned work will result in developments that make collective intelligence detection tools practical by providing early warning signs of shared misconceptions.

To attain these goals, the research will apply a general framework that incorporates (1) network models that help understand the social processes that lead to observed decision patterns; (2) machine learning tools that draw from uncovered processes to identify signals that optimize the accuracy of collective judgment; and (3) evaluation testbeds that use simulation tools in addition to rich high-dimensional, real-world data about the various stages and performance of group decisions. This research will contribute to societally-relevant outcomes, including: (a) understanding decision-making in online investment and lending settings to enhance the economic growth of underserved market segments; (b) generating novel knowledge about the performance benefits of collective judgment, and (c) quantifying the link between limited opinion diversity and crowd misconceptions. The project will connect undergraduate students, including women and under-represented minorities, to authentic practice in science and engineering research.

This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.

What is Collective Intelligence?

Concepts of collective intelligence (aka the “wisdom of crowds”) play increasingly fundamental roles in making judgments, for instance, about the future value of investments, patients’ reactions to therapies, how climate relates to immigration, what nations will financially default, outcomes of political races, or the replicability of scientific findings. Often, such complex decision-making scenarios violate the two key conditions that have been regarded by classic collective intelligence theories as necessary for the emergence of collective decisions that are superior to any individual’s personal judgment, namely the diversity of information among crowd members and their independence when forming opinions.

 

It is a challenging task to understand how groups can produce superior collective decision-making in case of complex problems, such as estimating the future value of investments, patients’ reactions to therapies, which nations will financially default, outcomes of political races, or which scientific findings can be replicated. Each of these settings challenge one or more of the three pillars of classic collective intelligence theory: (a) the independence of decision-makers, (b) opinion diversity, and (c) the suitability of a simple opinion aggregation mechanism, like averaging. Investigations of new expressions of crowd intellect under relaxed collective intelligence conditions have become possible due to recent Web technologies that record traces of various types of decision-making and knowledge production processes in large collectives.



How can we tap into the ‘Wisdom of Crowds’?

The goal of this research is to enhance collective intelligence and detect ahead of time misconceptions caused by, e.g., herding or group-think, through the development of models and algorithmic tools that elicit relevant crowd wisdom in high-stake real-world settings. In this project we design network-aware machine learning tools that elicit useful diversity, counter herding and homophily effects in restraining the wisdom of crowds, and improve the accuracy of collective forecasting. The project focuses mainly on online investments (i.e., crowdfunding), but parts of the work rely on other settings, such as reputation systems on sharing economy platforms, email communication networks, news media networks, and digital music consumption.

 

 To attain this goal, we develop a general framework that incorporates:

 

1) Network models that help understand the social processes that lead to observed decision patterns

 

2) Machine learning tools that draw from the uncovered processes to identify signals that optimize the accuracy of collective judgment

 

3) Evaluation testbeds that use simulation tools in addition to rich high-dimensional real-world data about the various stages and performance of group decisions

 

This framework contributes to the advancement of complex systems theory by predicting when will crowds provide accurate decision-making support for complex problems and when will they fail dismally. The research also helps identify robust collective intelligence signals and aids the development of opinion aggregation mechanisms that efficiently capitalize on diversity. Furthermore, the planned work will result in developments that make collective intelligence detection tools practical by providing early warnings of collective misconceptions.





This new framework enhances our understanding of the mechanisms that govern decision-making under social influence, which is an aspect that traditional collective intelligence theories fail to incorporate. This research contributes to several societally-relevant outcomes, such as:

 

1) Understanding decision-making in online investment- and lending settings to enhance the economic growth of under-served market segments

 

2) Generating novel knowledge about the performance benefits of collective judgments

 

3) Quantifying the link between limited opinion diversity and crowd misconceptions

 

Knowledge resulting from involved network models informs feature design and selection for a machine learning framework that contributes novel tools for diagnosing crowd wisdom and recommends strategies that optimize the efficiency of collective decisions.


Emoke-Agnes Horvat

Assistant Professor

School of Communication

Northwestern University

http://www.agneshorvat.info/






Henry Dambanemuya

Technology and Social Behavior

Northwestern University

https://www.dambanemuya.com/







Kyosuke Tanaka

Media, Technology and Society

Northwestern University

http://kyosuketanaka.com/





Yixue Wang

Technology and Social Behavior

Northwestern University

https://sites.northwestern.edu/yixue







Nick Hagar

Media, Technology and Society

Northwestern University

https://nhagar.github.io/







Igor Zakhlebin

Northwestern University

3Red Partners

https://www.igorzakhlebin.com/







Eunseo (Dana) Choi

MIT

  • H Dambanemuya and E-Á Horvát, A multi-platform study of crowd signals associated with successful online fundraising, Proceedings of the ACM (PACM) Human-Computer Interaction CSCW’21 (to appear)
   
  • E Choi and E-Á Horvát, Airbnb’s reputation system and gender differences among guests: Evidence from large-scale data analysis and a controlled experiment, Proceeding of the International Conference on Social Informatics, 2019

    http://link.springer.com/10.1007/978-3-030-34971-4_1

     
  • H Dambanemuya, M Joshi, E-Á Horvát, Network perspective on the efficiency of peace accords implementation, Proceedings of the IEEE/ACM International Conference on Advances in Social Network Analysis and Mining, Vancouver, Canada, 2019

    https://dl.acm.org/doi/10.1145/3341161.3342895

       

You can view the code developed as part of this project through our GitHub repository.