AI models overestimate smartness of people: Study
The rapid advancements in artificial intelligence (AI) have led to the development of sophisticated models that can process vast amounts of data, learn from experiences, and make decisions autonomously. However, a recent study by scientists at HSE University has revealed that these AI models, including popular ones like ChatGPT and Claude, tend to overestimate the smartness of people. This overestimation can lead to suboptimal decision-making and poor performance in strategic thinking games.
The study, which was conducted using the Keynesian beauty contest, a game that requires players to think strategically and make decisions based on their understanding of human behavior, found that AI models often play “too smart” and lose because they assume a higher level of logic in people than is actually present. This phenomenon has significant implications for the development of AI models and their application in real-world scenarios.
To understand the study’s findings, it’s essential to delve into the concept of the Keynesian beauty contest. The game, which was first introduced by John Maynard Keynes, is a thought experiment that involves a newspaper contest where participants are asked to choose the six most beautiful faces from a set of photographs. The winner is the person who chooses the face that is most popular among all the participants. The twist is that the winner is not the person who chooses the face that they think is the most beautiful, but rather the face that they think others will think is the most beautiful.
The Keynesian beauty contest is a classic example of a game that requires strategic thinking and an understanding of human behavior. It’s a game that is often used to illustrate the concept of common knowledge, which refers to the knowledge that is shared by all participants in a game. In the context of the Keynesian beauty contest, common knowledge refers to the understanding that all participants have about the behavior of others.
The study by HSE University scientists used the Keynesian beauty contest as a platform to test the performance of AI models, including ChatGPT and Claude. The researchers found that these models tended to overestimate the smartness of people and played “too smart” by assuming a higher level of logic in human behavior than was actually present. This led to suboptimal decision-making and poor performance in the game.
The study’s findings have significant implications for the development of AI models. If AI models are to be effective in real-world scenarios, they need to be able to understand human behavior and make decisions based on that understanding. However, if these models overestimate the smartness of people, they may make decisions that are not optimal.
One of the possible explanations for the AI models’ tendency to overestimate human smartness is that they are trained on data that is biased towards rational behavior. In other words, the data that is used to train these models is often generated by experts or individuals who are familiar with game theory and strategic thinking. As a result, the models learn to recognize and mimic this type of behavior, which may not be representative of the general population.
Another possible explanation is that AI models lack the ability to understand the cognitive biases and heuristics that influence human decision-making. Humans are prone to making mistakes and exhibiting biases in their decision-making, such as confirmation bias, anchoring bias, and availability heuristic. AI models, on the other hand, are designed to optimize performance and minimize errors, which can lead them to overestimate human smartness.
The study’s findings also have implications for the application of AI models in real-world scenarios. For example, in finance, AI models are often used to make investment decisions or predict market trends. However, if these models overestimate human smartness, they may make decisions that are not optimal or fail to anticipate market fluctuations.
In conclusion, the study by HSE University scientists has highlighted the importance of understanding human behavior and cognitive biases in the development of AI models. The finding that AI models tend to overestimate human smartness has significant implications for the application of these models in real-world scenarios. As AI continues to advance and become more pervasive in our lives, it’s essential to recognize the limitations of these models and develop more nuanced and human-centered approaches to AI development.
News Source: https://www.sciencedirect.com/science/article/abs/pii/S0167268125004470
About the Author
