‘The Peak of Stupidity’ and ‘The Valley of Despair’: HSE Economists Propose an Explanation for the Dunning–Kruger Effect

© iStock

The Dunning–Kruger effect, which describes a sharp surge in self-confidence among beginners followed by an equally rapid decline as they gain experience, can be explained by the nature of the learning process and the acquisition of new knowledge. This conclusion was reached by Andrey Vorchik of the HSE Faculty of Economic Sciences together with independent researcher Murat Mamyshev. They developed a mathematical model of learning and demonstrated how subjective confidence is formed and changes as knowledge accumulates, as well as how teachers can reduce the ‘valley of despair’ experienced by learners.

In 1999, American psychologists David Dunning and Justin Kruger showed that people with low levels of competence often overestimate their abilities. Beginners who are only starting to master a field of knowledge may make hasty judgements because of overconfidence. They quickly reach the so-called ‘peak of stupidity’—for example, we are all familiar with armchair analysts who believe they understand football perfectly.

Over time, people become aware of their own incompetence, their confidence drops sharply, and they find themselves in the ‘valley of despair.’ Only as experience accumulates does subjective self-assessment begin to correspond to actual knowledge. Experiments show that this effect can indeed be observed in a number of fields, such as politics, although it is less pronounced in areas connected with creativity.

Previously, psychologists put forward various hypotheses about the causes of this effect, including explanations linked to emotions. Andrey Vorchik, Research Assistant of the Laboratory for Experimental Economics and Finance (LEEF) at the HSE Faculty of Economic Sciences, and independent researcher Murat Mamyshev proposed a different approach. They developed a formal model that explains the mechanism of learning and shows how a person’s perception of the completeness of their knowledge changes over time.

According to the theoretical model they developed, every field of knowledge consists of a limited set of facts. The key parameter is awareness—that is, the number of facts whose existence a person is aware of at all. Each fact is characterised by two dimensions: whether the person knows that it exists and whether they have mastered it, meaning whether they can formulate it independently. As a result, four types of knowledge about a fact can be identified: an ‘unknown unknown,’ a ‘known unknown,’ a ‘known known,’ and an ‘unknown known.’

The authors of the model distinguish between objective competence and subjective confidence. Competence is defined as the proportion of mastered facts out of all the facts belonging to a particular field of knowledge. Confidence, meanwhile, depends on awareness and is calculated as the proportion of ‘known knowns’ among the information the person is actually aware of. As long as individuals do not realise how vast the field they are studying really is, they rely on subjective impressions. For example, a beginner football player may know nothing about the offside rule because they have not yet encountered it, and therefore their awareness will be lower. For them, offsides remain ‘unknown unknowns.’

Andrey Vorchik

‘In formal education there are grades and examinations that help students themselves, their teachers, and everyone involved to understand a person’s actual level of knowledge. In self-education, however, excessive confidence is almost inevitable simply because there is no one nearby who can explain how accurately you have understood the subject,’ said Andrey Vorchik.

The key result of applying the proposed model lies in the authors’ demonstration of how confidence in one’s knowledge is formed and changes over time. In the early stages of learning, awareness is low. Because of this, people feel that they have quickly mastered the main aspects of a new activity. For example, someone who is just learning to cook may memorise several recipes and believe they already understand the subject fairly well. Yet many important things—such as temperature control—have not yet appeared in their experience. For them, these remain ‘unknown unknowns’. This naturally leads to the ‘peak of stupidity.’ Later, however, learners begin to notice many previously unknown aspects. Their awareness expands dramatically, and because the amount of actual knowledge remains roughly the same, their confidence falls sharply. In learning to cook, for instance, a person may realise that the meat they have prepared is too tough, but not know how to control this. Facts related to the problem become ‘known unknowns’. This is how they enter the ‘valley of despair.’ At higher levels of competence, subjective self-assessment gradually comes closer to reality.

At the same time, avoiding the Dunning–Kruger effect is not easy. ‘The point is that the effect is a cognitive bias, and in many situations we will still experience excessive confidence. But if you know that the Dunning–Kruger effect exists and are able to recall it at the right moment, your chances of escaping the trap of overconfidence will be higher. Better still, you should ‘look around’ and actively seek encounters with things you do not yet understand,’ explained Andrey Vorchik.

The model also shows that people can fall into traps of self-confidence. If someone uses subjective confidence as a measure of success, they may stop learning at an early stage, avoiding situations in which their confidence declines. This helps explain why superficial knowledge remains so widespread even in an age of readily available information.

Thus, the study demonstrates that the Dunning–Kruger effect may arise as a natural consequence of the learning process itself, without the need to invoke additional psychological mechanisms. This opens up new possibilities for designing educational programmes and learning tools.

‘Excessive confidence can be avoided if people always go a little further and seek out something they do not understand. In my view, teachers should not oversimplify complex subjects too much. It is important to discuss the connections between facts within a field and across different disciplines; to talk about paradoxes and misconceptions; to ask difficult and, I would even say, intriguing questions. And of course, to point out mistakes. All of this helps create a sense of puzzlement, which contributes so greatly to progress in both science and learning,’ Andrey Vorchik noted.