Early Detection of Collective Misconceptions with Network-Aware Machine Learning Tools
NSF Award # 1755873 (March 15, 2018 – February 28, 2021)
This project builds theory, algorithms, and frameworks that can be used to design network-aware machine learning tools aimed at eliciting useful diversity and improving the accuracy of collective forecasting. Researchers in the social and economic sciences know that there is great capacity for collective intelligence to emerge from Web-based systems. Yet herding and homophily effects often restrain the wisdom of crowds, vastly limiting this potential. The research furthers the study of complex systems by introducing a new framework that improves our understanding of the mechanisms that govern decision-making under social influence. Advancing complex systems theory in this way greatly enhances the ability to predict when crowds will provide accurate decision-making support for complex problems and when they will fail miserably. Further, the research aids the development of opinion aggregation mechanisms that efficiently capitalize on diversity. The planned work will result in developments that make collective intelligence detection tools practical by providing early warning signs of shared misconceptions.
To attain these goals, the research will apply a general framework that incorporates (1) network models that help understand the social processes that lead to observed decision patterns; (2) machine learning tools that draw from uncovered processes to identify signals that optimize the accuracy of collective judgment; and (3) evaluation testbeds that use simulation tools in addition to rich high-dimensional, real-world data about the various stages and performance of group decisions. This research will contribute to societally-relevant outcomes, including: (a) understanding decision-making in online investment and lending settings to enhance the economic growth of underserved market segments; (b) generating novel knowledge about the performance benefits of collective judgment, and (c) quantifying the link between limited opinion diversity and crowd misconceptions. The project will connect undergraduate students, including women and under-represented minorities, to authentic practice in science and engineering research.
This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.
Project Overview
What is Collective Intelligence?
Concepts of collective intelligence (aka the “wisdom of crowds”) play increasingly fundamental roles in making judgments, for instance, about the future value of investments, patients’ reactions to therapies, how climate relates to immigration, what nations will financially default, outcomes of political races, or the replicability of scientific findings. Often, such complex decision-making scenarios violate the two key conditions that have been regarded by classic collective intelligence theories as necessary for the emergence of collective decisions that are superior to any individual’s personal judgment, namely the diversity of information among crowd members and their independence when forming opinions.
It is a challenging task to understand how groups can produce superior collective decision-making in case of complex problems, such as estimating the future value of investments, patients’ reactions to therapies, which nations will financially default, outcomes of political races, or which scientific findings can be replicated. Each of these settings challenge one or more of the three pillars of classic collective intelligence theory: (a) the independence of decision-makers, (b) opinion diversity, and (c) the suitability of a simple opinion aggregation mechanism, like averaging. Investigations of new expressions of crowd intellect under relaxed collective intelligence conditions have become possible due to recent Web technologies that record traces of various types of decision-making and knowledge production processes in large collectives.
How can we tap into the ‘Wisdom of Crowds’?
The goal of this research is to enhance collective intelligence and detect ahead of time misconceptions caused by, e.g., herding or group-think, through the development of models and algorithmic tools that elicit relevant crowd wisdom in high-stake real-world settings. In this project we design network-aware machine learning tools that elicit useful diversity, counter herding and homophily effects in restraining the wisdom of crowds, and improve the accuracy of collective forecasting. The project focuses mainly on online investments (i.e., crowdfunding), but parts of the work rely on other settings, such as reputation systems on sharing economy platforms, email communication networks, news media networks, and digital music consumption.
To attain this goal, we develop a general framework that incorporates:
1) Network models that help understand the social processes that lead to observed decision patterns
2) Machine learning tools that draw from the uncovered processes to identify signals that optimize the accuracy of collective judgment
3) Evaluation testbeds that use simulation tools in addition to rich high-dimensional real-world data about the various stages and performance of group decisions
This framework contributes to the advancement of complex systems theory by predicting when will crowds provide accurate decision-making support for complex problems and when will they fail dismally. The research also helps identify robust collective intelligence signals and aids the development of opinion aggregation mechanisms that efficiently capitalize on diversity. Furthermore, the planned work will result in developments that make collective intelligence detection tools practical by providing early warnings of collective misconceptions.
This new framework enhances our understanding of the mechanisms that govern decision-making under social influence, which is an aspect that traditional collective intelligence theories fail to incorporate. This research contributes to several societally-relevant outcomes, such as:
1) Understanding decision-making in online investment- and lending settings to enhance the economic growth of under-served market segments
2) Generating novel knowledge about the performance benefits of collective judgments
3) Quantifying the link between limited opinion diversity and crowd misconceptions
Knowledge resulting from involved network models informs feature design and selection for a machine learning framework that contributes novel tools for diagnosing crowd wisdom and recommends strategies that optimize the efficiency of collective decisions.
Research Team
Emoke-Agnes Horvat
Assistant Professor
School of Communication
Northwestern University
Henry Dambanemuya
Technology and Social Behavior
Northwestern University
Kyosuke Tanaka
Media, Technology and Society
Northwestern University
Yixue Wang
Technology and Social Behavior
Northwestern University
https://sites.northwestern.edu/yixue
Nick Hagar
Media, Technology and Society
Northwestern University
Igor Zakhlebin
Northwestern University
3Red Partners
https://www.igorzakhlebin.com/
Eunseo (Dana) Choi
MIT
Publications
- H Dambanemuya and E-Á Horvát, A Multi-platform Study of Crowd Signals Associated with Successful Online Fundraising.Proceedings of the ACM (PACM) Human-Computer Interaction CSCW’21. https://dl.acm.org/doi/10.1145/3449189
- K Tanaka and E-Á Horvát, (Un)intended consequences of networking in individual and network-level efficiency, Applied Network Science 4: 77, 2019 https://link.springer.com/epdf/10.1007/s41109-019-0196-2
- E Choi and E-Á Horvát, Airbnb’s reputation system and gender differences among guests: Evidence from large-scale data analysis and a controlled experiment, Proceeding of the International Conference on Social Informatics, 2019 http://link.springer.com/10.1007/978-3-030-34971-4_1
- H Dambanemuya and E-Á Horvát, Harnessing collective intelligence in P2P lending, Proceedings of ACM Web Science, Boston, 2019 http://dl.acm.org/citation.cfm?doid=3292522.3326040
- I Zakhlebin and E-Á Horvát, Investor retention in equity crowdfunding, Proceedings of ACM Web Science, Boston, 2019 http://doi.org/10.1145/3292522.3326037
- H Dambanemuya, M Joshi, E-Á Horvát, Network perspective on the efficiency of peace accords implementation, Proceedings of the IEEE/ACM International Conference on Advances in Social Network Analysis and Mining, Vancouver, Canada, 2019 https://dl.acm.org/doi/10.1145/3341161.3342895
- Y Wang and E-Á Horvát, Gender differences in the global music industry: Evidence from MusicBrainz and The Echo Nest, Proceedings of AAAI Conference on Web and Social Media, Munich, Germany, 2019 https://ojs.aaai.org/index.php/ICWSM/article/view/3249
- K Tanaka and E-Á Horvát, Networking strategies and efficiency in human communication networks, Proceedings of the International Workshop on Complex Networks and their Applications, pp. 258-270, 2019 https://link.springer.com/chapter/10.1007%2F978-3-030-05414-4_21
- E-Á Horvát, J Wachs, R Wang, and A Hannák, The role of novelty in securing investors for equity crowdfunding campaigns, Proceedings of the AAAI Human Computation and Crowdsourcing, pp. 50-59, 2018 https://ojs.aaai.org/index.php/HCOMP/article/view/13336/13184
- N Hagar, J Wachs, and E-Á Horvát, Publishing patterns reflect political polarization in news media(preprint: https://arxiv.org/abs/2101.05044)