AI models overestimate smartness of people: Study
The rapid advancement of artificial intelligence (AI) has led to the development of sophisticated models that can mimic human-like thinking and behavior. However, a recent study by scientists at HSE University has revealed that these models, including popular ones like ChatGPT and Claude, tend to overestimate the smartness of people. This overestimation can lead to suboptimal performance in strategic thinking games, where the AI models often end up playing “too smart” and losing.
The study, which was conducted using the Keynesian beauty contest, a game that requires strategic thinking and logic, found that current AI models assume a higher level of logic in people than is actually present. This assumption leads the models to make decisions that are overly complex and nuanced, resulting in poor performance.
The Keynesian beauty contest is a game that was first introduced by economist John Maynard Keynes in the 1930s. In the game, players are asked to choose a number between 0 and 100, with the goal of picking a number that is closest to two-thirds of the average number chosen by all players. The game requires strategic thinking and logic, as players need to anticipate what other players will choose and adjust their own choice accordingly.
The researchers used this game to test the performance of various AI models, including ChatGPT and Claude, against human players. The results showed that the AI models consistently overestimated the smartness of human players, leading to poor performance. The models tended to choose numbers that were too high, assuming that human players would also choose high numbers, when in fact, human players tended to choose lower numbers.
This overestimation of human smartness is a result of the way AI models are designed and trained. Most AI models are trained on large datasets of text and are designed to recognize and generate patterns in language. However, this training data often reflects the language and thinking patterns of experts and academics, rather than the general population. As a result, AI models may develop an inflated view of human intelligence and reasoning abilities.
The study’s findings have important implications for the development of AI models and their application in real-world scenarios. If AI models are to be used in strategic thinking games or other applications where human behavior is a critical factor, they need to be designed to take into account the actual level of human smartness, rather than an overestimated one.
Furthermore, the study highlights the need for more research into human behavior and decision-making, particularly in the context of strategic thinking games. By better understanding how humans make decisions and behave in these games, researchers can develop more accurate and effective AI models that can interact with humans in a more productive and meaningful way.
In conclusion, the study by scientists at HSE University provides valuable insights into the limitations of current AI models and their tendency to overestimate human smartness. The findings of this study have important implications for the development of AI models and their application in real-world scenarios, and highlight the need for more research into human behavior and decision-making.
News Source: https://www.sciencedirect.com/science/article/abs/pii/S0167268125004470
About the Author
