The social disincentives of warning about unlikely risks
If you knew about a potential large-scale risk that, although unlikely, could kill millions, would you warn society about it? You might say yes, but many people are reluctant to warn.
In ten studies, Matt Coleman, Joshua Lewis, Christoph Winter, and I explored a psychological barrier to warning about low-probability, high-magnitude risks.
In short, we found that people are reluctant to warn because they could look bad if the risk doesn’t occur. And while unlikely risks probably won’t happen, they should still be taken seriously if the stakes are large enough. For example, it’s worth wearing a seat belt because, even though a car crash is unlikely, its consequences would be so severe. Unfortunately, reputational incentives are often not aligned with what’s most beneficial for society. People would rather keep quiet and hope nothing happens rather than be seen as overly alarmist.
Below, I summarize some of our studies, discuss the underlying psychology of the phenomenon, and suggest possible strategies to encourage risk warning in society. If you want more information about the studies, you can check out our research paper (including all data, materials, scripts, and pre-registrations).
Reputational fears of warning about unlikely risks
In Study 1, we asked 397 US online participants to imagine they were biological risk experts and believed there was a 5% chance of a new, extremely dangerous virus emerging within the next three years. They could warn society about the risk and recommend a $10-billion investment to develop a vaccine that would prevent all possible harm. If no vaccine is developed and the virus emerges, it will kill millions of people and lead to billions of dollars of economic damage. But if the virus doesn’t emerge, nobody will be harmed, and the money invested in developing the vaccine will have been wasted. Participants were then asked how likely or unlikely they would be to warn society about the risk, and how concerned they would be that society would blame them for warning about the virus.
We hypothesized that people would be reluctant to warn about an unlikely risk due to fear of blame. If true, they should be more willing to warn anonymously. So, we told half the participants their identity would be public if they warned, and the other half that their identity would remain anonymous. We assured both groups that key decision-makers would take their warnings seriously. As expected, participants were less likely to warn society about the risk publicly than anonymously (M = 4.56 vs. M = 5.25, on a scale from 1 to 7, p < .001). And they were more concerned about being blamed for warning about the risk publicly than anonymously (M = 4.12 vs. M = 2.90, p < .0001).
Warning disincentives are specific to unlikely risks
If you warn about a low-probability risk, the most likely outcome is that the risk won’t materialize, and you’ll look naive or overly alarmist. In contrast, if you warn about a high probability risk, the risk probably will materialize, and you’ll look smart. Thus, we hypothesized that people would be particularly reluctant to publicly warn about unlikely risks compared to likely ones since for the latter, they know that their prediction will probably turn out to be true.
To test this, in Study 2a, 539 US participants imagined they believed that there was an extremely damaging storm that could emerge within the next three months. They were planning to warn society about the risk and recommend that the government invest in minimizing harm from the storm. To test our hypothesis, we randomly varied the likelihood and severity of the storm. It had either a 1% chance of killing 100 million people or a 99% chance of killing one million people. This way, there was roughly the same expected harm (multiplying the probability and the number of deaths). Participants were told that by default, they would warn society about the storm publicly, disclosing their identity to everyone. But if they wanted, they could pay to instead warn society anonymously. Participants indicated how willing they would be to pay a quarter of their monthly income to warn anonymously instead of publicly.
As expected, participants were more willing to incur personal financial costs to warn anonymously instead of publicly about an unlikely than likely risk (M = 3.67 vs. M = 3.24, p < .0001). In line with that, participants feared blame more for warning about the unlikely risk than the likely risk (M = 4.39 vs. M = 3.65, p < .0001).
This finding also suggests that people want to help but are just inhibited by their reputational concerns. And if there’s a way to help without getting blamed, they will do it even at a personal financial cost.
Reluctance to warn publicly is stronger in China
Next, we tested whether the findings generalize across cultures. For that, we translated the previous study into Mandarin and conducted the same study with 381 Chinese participants (Study 2b).
We found that Chinese participants, in comparison to US participants, were more willing to pay a quarter of their monthly income to warn anonymously instead of publicly (p < .0001) (both for likely and unlikely risks). A possible explanation is that China is more collectivist than the US, which may drive them to be more concerned about what others think of them. Indeed, we found that Chinese participants were particularly concerned about blame. More research is needed to investigate the robustness of and mechanisms behind this cross-cultural effect.
Policymakers are reluctant to warn
So far, we tested people from the general public. Next, we tested groups with decision-making power and insights into societal issues.
In Study 3, 498 elected US local government policymakers imagined they were the only one who knew about a risk to their municipality’s revenue. To test our hypothesis, they were either told there was a 1% chance of a $100 million revenue loss or a 99% chance of a $1 million revenue loss (again, these risks had similar expected value). Participants then imagined that they would alert the public to the threat through an op-ed in the local newspaper. Importantly, they could decide whether to publish it anonymously or under their name. As expected, policymakers favored anonymity more for warning about unlikely, severe risks compared to more likely, less severe risks (M = 3.87 vs. M = 4.14, on a scale from 1 to 5, p < .0001).
AI researchers fear criticism for raising alarms about unlikely risks
Next, we turned to the risks related to artificial intelligence. In Study 4, 500 AI researchers (authors from top-tier machine learning conferences) were asked to consider risks from advanced AI, such as job losses, AI suffering, and catastrophic events. They imagined warning society and the government about these risks over the next five years. Two types of AI risks were evaluated: one unlikely but severe, and one likely but moderately severe. Both risks were considered equally important to prepare for. As expected, AI researchers were more worried about being criticized for warning about the unlikely but major AI-related risk than for warning about the likely but smaller-scale risk (M = 3.56 vs. M = 2.96, p < .001).
Anticipating others’ outcome bias
Are people right to be concerned about getting blamed? Unfortunately, yes. That’s because of outcome bias — the tendency to evaluate the quality of a decision based on the outcome instead of whether the best option was chosen given the then-available information. For example, suppose you could enter a die-roll gamble that gets you $10 if the number four shows up and otherwise requires you to pay a dollar. Should you accept the gamble? Yes. Even though you win in only one out of six cases, the reward is large enough that you can accept the likely $1 loss. But what if you accept the gamble and lose? You may wish you hadn’t accepted the gamble. That’s outcome bias at play. Of course, you knew all along that you’d probably lose. And then you did. And yet, it was the correct decision.
When other people incur costs based on your warning, those people’s outcome bias is of particular concern. If the risk in your warning fails to materialize, they might blame you even if the decision was justified based on the evidence available at the time. Unfortunately, this is never more true than with large societal risks. It requires many societal resources to manage them, and people get very upset when these resources appear wasted.
We found that people tend to judge the decision to warn about a risk as less justifiable if the risk did not occur than if it did. To make matters worse, we discovered this effect in a sample of judges and lawyers — those on whom we depend most to fairly judge people's behavior. (For more details, see Study 8 in our paper.)
Strategies to encourage warnings
Given our understanding of the social disincentives against warning about unlikely, large-scale risks, what could we do to encourage warning?
Anonymous warning systems
As we saw above, when people can warn about risks anonymously, they are much more willing to warn even about unlikely risks. This suggests a potential policy proposal. Where feasible, policymakers could set up institutions allowing experts to anonymously express their concerns about potential large-scale societal risks. Their risk estimates, arguments, and evidence would be publicized, including information about their expertise and perhaps also the type of organization they work for (similar to whistleblowing). Crucially though, their identity would be kept secret. Such systems may not always be feasible, but when low-probability risk warnings are most valuable, their feasibility should be explored and their effectiveness tested.
Routine risk assessment prompts
Imagine if institutions implemented a routine protocol requiring experts to identify and evaluate potential risks on a regular basis. In this process, experts would be prompted to detail every risk they perceive, even unlikely and speculative ones, provide their assessments of these risks, and offer recommendations for addressing them. By formalizing this process, it could become standard practice for experts to communicate risks, reducing their apprehension about being blamed since they're simply responding to a request for their expertise.
Risk communication
What can experts do when no specific risk warning systems are in place?
Provide objective evidence. In Study 7, we found that people are more likely to warn if there is objective evidence for the risk. For example, if you warn about an asteroid heading towards Earth and that asteroid is relatively easily visible in the sky, people won’t blame you if the risk ultimately doesn’t materialize. After all, you’ve simply drawn people’s attention to an unambiguous risk. And people can’t question your judgment because it’s so obvious that the risk was a real possibility. Unfortunately, often, it’s not possible to have unambiguous objective evidence. Instead, risk estimates are often based on the subjective estimates of experts or conditional predictions of models, perhaps based on a vast number of difficult-to-follow arguments and analyses. In such cases, people have to trust your judgment, and blame is harder to escape.
Clarify the risk estimate. In Study 9, we found that when warners clarify that they think the risk is unlikely but worthwhile to address, they limit reputational backlash. People considered a warner of risk that didn’t occur as more trustworthy and less blameworthy if the warner specified the likelihood of the risk than if not.
Overcome the first-warner hurdle. Study 5 revealed that people are much more likely to warn about unlikely large-scale risks if others have already warned. That’s because they know that they won’t be the only ones getting blamed if their prediction turns out wrong. That means that the ones who warn first have it the hardest. But it also means that your impact will be most significant if you’re amongst the first. Your warning will make it easier for others to speak up, too. This also means that there’s an advantage to experts having different beliefs about the risk likelihood. Those who believe a certain risk is very likely (or are particularly willing to incur reputational costs) are more likely to warn. In a simulation (Study 6), we showed that this can initiate a warning cascade. For example, thanks to experts like Geoffrey Hinton, it is now easier to warn about the potential risks of misaligned artificial intelligence. It’s also a reason why public risk warning letters with many co-signers can be effective.
Should we listen to everyone who’s warning about risks?
What about the conspiracy theories and culty doomsday predictions that always turn out false? Aren’t there people who use scaremongering as a tactic to get attention? Clearly, we can’t take everyone equally seriously. Someone who has expertise, who provides reasons and evidence, and who really tries to be intellectually honest, modest, and open-minded should be taken more seriously than a non-expert who makes wild, overconfident claims without even trying to provide sensible arguments. Indeed, sometimes, a warner who admits the risk is speculative and unlikely should be taken more seriously than someone who appears to be sure the risk will occur. In fact, this is another reason to be attentive to low-probability risk warnings. When have you heard a doomsday cult propose a 1% chance of the world ending on a given date?
If a warner is genuine and epistemically virtuous, it’s important not to ignore or dismiss them too quickly, even if you disagree with them and think their predictions are off. At least someone should take a minute to see if they may have a point. That requires trying to understand and engage with their arguments so that we can have an honest debate and figure out whether and how to address the risk in a reasonable way. And if we decide to invest in precautionary safety measures, it will, in retrospect, often seem as if these resources were wasted. That’s precisely what we should expect to happen if we’re doing things right. A wise and robust society understands this.
Finally, let’s not ridicule people who warn sincerely and with good intentions. Apart from being mean, it’s reckless. As we saw, it creates social disincentives against warning about unlikely risks. And many of the biggest risks may be rather unlikely. To manage these appropriately, we need a culture that makes it easy for experts to warn; that means we need to encourage and not discourage warners. So, at the very least, let’s appreciate their efforts to help. It probably took courage.
Further readings
Caviola, L., Coleman, M. B., Winter, C., & Lewis, J. (2024, June 14). Crying wolf: Warning about societal risks can be reputationally risky.
Dorison, C. (2023). Reputational Rationality Theory.
Scott Alexander. (2020). A Failure, But Not of Prediction. Slate Star Codex
Kat Woods. The Parable of the Boy Who Cried 5% Chance of Wolf. EA Forum.
Thanks to Stefan Schubert, Josh Lewis, Carter Allen, and Matt Coleman for their comments and to Julian Jamison for suggesting the routine risk assessment prompts idea.
Please share your questions or suggestions in the comments. Do you know of any other barriers that hinder responsible management of large-scale risks, along with potential solutions?
This is really interesting. Do you think these results would be less relevant in domains where the social incentives to warn (e.g., fame, self-aggrandizement of one’s own work) significantly outweigh the disincentives (fear of blame)? I had in mind AI risk as plausibly being one such domain.