From the Inaugural Address of President John F. Kennedy, January 20, 1961
“The world is very different now. For man holds in his mortal hands the power to abolish all forms of human poverty and all forms of human life.”
Max Tegmark of MIT’s Future of Life Institute is far more succinct. In his book Life 3.0, he contends that “Technology is giving life the potential to flourish like never before—or to self-destruct.”
Krauthammer and NASA on the Great Filter Theory
One of the goals of this book is to bring to the attention of the reader key concepts that are not widely discussed, which we believe are essential to understanding our 21st-century world. One such concept is the Great Filter Theory that was first brought to our attention in December 2011 by conservative columnist Charles Krauthammer, who in this Op Ed introduced us to the Fermi Paradox and the Drake Equation.
The great scientist Enrico Fermi came back to his office after lunch one day and asked, Where is everybody? Fermi thought that after 13.8 billion years since the Big Bang, our vast universe should be teeming with life. The Drake equation attempts to quantify the number of advanced civilizations that should exist.
According to Krauthammer:
Modern satellite data, applied to the Drake Equation, suggest that the number should be very high. So why the silence? Carl Sagan (among others) thought that the answer is to be found, tragically, in the final variable: the high probability that advanced civilizations destroy themselves.
In other words, this silent universe is conveying not a flattering lesson about our uniqueness but a tragic story about our destiny. It is telling us that intelligence may be the most cursed faculty in the entire universe — an endowment not just ultimately fatal but, on the scale of cosmic time, nearly instantly so.” For more information on the Fermi Paradox, click here.
We later learned that Krauthammer is referring to the Great Filter Theory.
The idea is that before any civilization can become a spacefaring one, that civilization would need to overcome certain hurdles or filters. Great Filter theories, which were first proposed by economist Robin Hanson, suggest that there is some insurmountable evolutionary hurdle that prevents most life from reaching technological civilization. This filter could be behind us (we are the lucky ones who made it through) or ahead of us (civilizations tend to destroy themselves before becoming interstellar). For more information on the Great Filter Theory, click here.
Several NASA scientists explored this theory as well in a paper entitled Avoiding the ‘Great Filter’: Extraterrestrial Life and Humanity’s Future in the Universe. The paper explores the Fermi Paradox and the Great Filter hypothesis to address why no extraterrestrial civilizations have been detected despite the statistical likelihood of life in the ~two trillion galaxies in our universe. It argues that a “Great Filter”—a barrier preventing civilizations from reaching interstellar communication or travel—may explain this silence, with existential risks like nuclear war, climate change, pandemics, and artificial intelligence (AI) as potential filters humanity must navigate to survive. The scientists wrote:
The key to humanity successfully traversing such a universal filter is found in understanding what characteristics the barrier will constrain, identifying those attributes in ourselves, and neutralizing them in advance. Human civilization over the past 5000+ years, and in particular since 1945, has revealed much of what would surely impede, if not outright arrest, our aspirations to colonize other worlds in the Solar System and beyond.
It seems as though nearly every great discovery or invention, while pushing back the borders of our technological ignorance, is all too quickly and easily turned to destructive ends. Examples such as splitting the atom, biomedical innovations, and resource extraction and consumption come to mind with disconcerting swiftness.” –from Page 4 of the report
The authors propose that humanity is in a “Window of Peril” that began with the development of nuclear weapons in 1945, marking our capacity for self-destruction. They emphasize collaboration and introspection to overcome destructive tendencies (e.g., racism, inequity) and advance toward a Kardashev Type I civilization, capable of harnessing all planetary energy. Using historical data on computing power and space exploration, they model potential timelines for human expansion, suggesting interplanetary colonization by the end of this century and interstellar missions by the 23rd century, which could ensure survival by making humanity a multi-world species.
“If life arising on Earth is ever to know of life elsewhere, assuming such exists, we as the Earth’s sole technological species must first come fully to terms with ourselves and our environment. The struggle for survival, security and dominance – all rooted in human passions – drives creativity and with it, civilization and invention [13]. As history has shown time and again, however, this cleverness comes at great cost. The human brain, still orders of magnitude more complex in terms of synaptic connections than the transistor-based structures underlying the most advanced supercomputers, holds the key. Using our demonstrated inventiveness to proactively recognize, diagnose and formulate countermeasures to the most serious threats to our existence, humanity may yet avert the Great Filter.”— Page 4 of the report.
They conclude that by fostering global cooperation and technological stewardship, humanity can surpass these filters, potentially becoming the first civilization to achieve interstellar presence, thus answering the Fermi Paradox’s question: “Where is everybody?”
“In so doing we would likely emerge downstream of the Great Filter as a near Type I civilization on the Kardashev scale [14], ready to seek our place in a future greater than what we could realize if confined to just our home world. Indeed, recent modeling suggests human-crewed exploration of our Solar System beyond Mars may well be possible within this century [15], [16]. Analysis of these leading threats has found them to include large-scale nuclear warfare, pathogens (both naturally occurring and engineered), artificial intelligence, impacts from asteroids and climate change [17].“ – From Page 4 of the report
While the Great Filter Theory provides interesting context for our discussion, the most important thing to understand is that humanity faces several unprecedented existential threats, threats that have not been sufficiently considered, as the NASA authors suggest.
Sir Martin Rees: 2003 Book Our Final Hour and the Creation of CSER
Discussing the end of humanity is not an easy conversation, which is why it might take a cosmic perspective to really embrace this topic. No global thinker has done more to focus humanity’s attention on the need to address the existential threats facing our species than Great Britain’s Royal Astronomer and former President of the Royal Society of Science, Sir Martin Rees.
At a time when there were more academic studies published on snowboarding than existential risk, Sir Martin looked around at the accelerating pace of scientific and technological change and issued a stark warning explaining that our global society needs to come up with a better way to measure its dramatic technological progress as well as the risks and rewards created by that progress.
In his book Our Final Hour (2003), Rees presents a compelling examination of the existential risks facing humanity in the early 21st century. Like NASA, he argued 20 years earlier that advances in science and technology, while offering immense benefits, also posed unprecedented threats that put our human civilization at risk of extinction. He identifies four civilizational threats, ranging from biotechnology/bioterrorism to AI to nuclear war to environmental degradation.
He then goes on to emphasize another key 21st-century concept we will take up in the next chapter, which is how profoundly unique the period we are all living through is today. Indeed, he argues that our period is the most important era since the Big Bang. Rees writes:
“It may not be absurd hyperbole—indeed, it may not even be an overstatement—to assert that the most crucial location in space and time (apart from the big bang itself) could be here and now. I think the odds are no better than fifty–fifty that our present civilization on Earth will survive to the end of the present century. Our choices and actions could ensure the perpetual future of life (not just on Earth, but perhaps far beyond it, too). Or in contrast, through malign intent, or through misadventure, twenty-first-century technology could jeopardize life’s potential, foreclosing its human and posthuman future. What happens here on Earth, in this century, could conceivably make the difference between a near eternity filled with ever more complex and subtle forms of life and one filled with nothing but base matter. “(Source: Chapter 1, page 21 in the electronic book.)
To navigatetheserisks, Rees advocates forimprovedglobalcollaboration and long-term thinking to manage and regulate emerging technologies, ethical responsibility, particularly for scientists and technologists who should consider the broader implications for their work, and improved public engagement, raising awareness and fostering informed public discourse on existential risks. At the time, Rees wrote, “I think the odds are no better than fifty–fifty that our present civilization on Earth will survive to the end of the present century.”
While that particular statement may have been meant as an urgent wake-up call, the fundamental point made by Sir Martin in this book and subsequent books is that our hyperconnected, high-tech world is profoundly fragile and if we are to survive this century well we will need to take focused global action.
Perhaps, most importantly, Sir Martin Rees did not simply write his book hoping that others would pick up the slack and generate the needed global dialogue and institutions to address existential risk; he went one step further and created the Centre for the Study of Existential Risk (CSER) to promote the needed dialogue.
Founded in 2012 by Rees, in collaboration with Huw Price, Philosopher of Science at Cambridge, and Jaan Tallinn, a co-founder of Skype at the University of Cambridge, CSER was set up to “study existential and global catastrophic risks and foster a worldwide community dedicated to their mitigation.” You can learn more about their wonderful programs here.
Lord Rees did not stop there. He helped his friend James Martin set up the Oxford Martin School for the 21st Century at Oxford. Martin, a futurist, gave the university its largest donation in history, $150 million, to start the Oxford Martin School.
As we will see in future chapters, perhaps just as important as his contribution to Oxford was the book Martin wrote, The Meaning of the 21st Century: A Vital Blueprint for Ensuring Our Future. In the book, James Martin calls for the creation of a Transition Generation that will help us navigate past our great filter to the world of Super Abundance, available to us only if we learn to get along.
We believe that Sir Martin Rees’s work in the UK has been the catalyst for the creation of a worldwide network of universities across the world taking up the serious-minded study of existential risk.
Rees’s co-founder at CSER, Jaan Tallinn, helped Max Tegmark create the Future of Life Institute at MIT. Indeed, we believe that Lord Rees’s book and commitment to encouraging humanity to engage with our existential challenges have led to the creation of an all-important X-Risk ecosystem of universities and think tanks that have been set up to address this challenge. We believe that the educational enrichment program we have set up has been the first group to pull together a catalogue of the various groups now working to mitigate existential risk, which you can access here.
(Please note that our catalogue does not yet include the University of Chicago’s Existential Risk Laboratory, whose mission is to understand and mitigate existential and global catastrophic risks while training the next generation of scholars and policymakers. Their work is focused on preventing events that could result in hundreds of millions of deaths or permanently curtail civilization’s long-term potential. They study four sets of risks: AI, Biosecurity, Nuclear Weapons, and Climate Change. See also this article from The New Yorker)
The Bulletin of Atomic Scientists
No discussion of existential risk can be complete without discussing the Bulletin of Atomic Scientists, the group created by Robert Oppenheimer and Albert Einstein, among others, to provide humanity with an annual assessment of how close we are all to self-annihilation.
The group was founded in 1945 by scientists concerned about the implications of nuclear weapons after the bombings of Hiroshima and Nagasaki. Initially a newsletter called Atomic Scientists of Chicago, the Bulletin evolved into a magazine to educate the public on the risks of nuclear technology and other global threats. Its mission is to inform policymakers, scientists, and the public about existential risks— primarily nuclear, but later expanding to climate change, biosecurity, and other emerging technologies—while advocating for responsible scientific and policy decisions.
The organization engages in public outreach through events, podcasts, and its website. Its most famous contribution is The Doomsday Clock, which is a symbolic representation of how close humanity is to global catastrophe.
The Doomsday Clock was introduced in 1947 by the Bulletin as a metaphor to illustrate the risk of global annihilation, initially focused on nuclear war. Designed by artist Martyl Langsdorf, the clock’s time is set by the Bulletin’s Science and Security Board with input from its Board of Sponsors, including Nobel laureates. Midnight represents global catastrophe, and the minutes or seconds to midnight indicate the perceived proximity to disaster.
The clock’s setting is updated annually based on factors like nuclear proliferation, geopolitical tensions, climate change, and technological risks. The Science and Security Board evaluates global events, scientific advancements, and policy decisions to determine whether to move the clock forward, backward, or keep it steady.
Closer than ever: The clock is now 89 seconds to midnight On January 25, 2025, the Board voted to move the clock one second closer to midnight, to 89 seconds.
“In 2024, humanity edged ever closer to catastrophe. Trends that have deeply concerned the Science and Security Board continued, and despite unmistakable signs of danger, national leaders and their societies have failed to do what is needed to change course. Consequently, we now move the Doomsday Clock from 90 seconds to 89 seconds to midnight—the closest it has ever been to catastrophe. Our fervent hope is that leaders will recognize the world’s existential predicament and take bold action to reduce the threats posed by nuclear weapons, climate change, and the potential misuse of biological science and a variety of emerging technologies.
“In setting the Clock one second closer to midnight, we send a stark signal: Because the world is already perilously close to the precipice, a move of even a single second should be taken as an indication of extreme danger and an unmistakable warning that every second of delay in reversing course increases the probability of global disaster.”
(You can learn more about The Bulletin here).
As the Bulletin makes clear, there can be no doubt that humanity is inching closer and closer to self-annihilation. Even more worrisome is the fact that we do not believe they have fully accounted for the new existential risks emanating from AI.
The Probability of Existential Risk from AI is 15% to 20%: Do We Need a Pause?
As we saw in the last chapter, the ChatGPT moment dramatically accelerated the arrival of AGI and ASI, with many forecasting that they will arrive before the end of the decade. There is no way humanity is prepared for a new autonomous digital superintelligence that is increasingly more powerful.
As we saw in the last two chapters, the current configuration of our new digital species is made up of Perception AI, Gen AI, and emerging Agentic AI and Robotic AI. Today’s AI are all still human-controlled. These AI can take us 85% to 100% of the way forward to create the economic benefits of a world of Super Abundance. There is virtually no risk of human extinction resulting from this cohort of AI.
The existential risk begins to emerge and accelerate once we begin to achieve Autonomous Superintelligence as our digital species acquires its own agency and becomes more autonomous. The fundamental question for AGI and ASI is: can we design them so that they are human-friendly and that they use their enormous powers for the good of humanity?
The short, honest answer is: no one knows, and the longer answer is: many are concerned. We now fully understand the call for the six- month pause Elon Musk and Max Tegmark made in 2023, along with 30,000 scientists, as it is very clear that we are not ready for autonomous AGI/ASI.
Among prominent futurists, Eliezer Yudkowsky stands out as arguably the most concerned about the existential risks posed by artificial intelligence, putting the chances of existential risk to our species at 70% to 90%.
Yudkowsky, a co-founder of the Machine Intelligence Research Institute (MIRI), has consistently warned that superintelligent AI could lead to human extinction if not properly aligned with human values. His concerns are rooted in the unpredictability of AGI’s goals and the difficulty of ensuring its control, emphasizing scenarios where AI could unintentionally cause catastrophic outcomes due to misaligned objectives or rapid self-improvement. For example, he has argued that even a 1% chance of existential catastrophe is unacceptably high given the stakes, advocating for extreme caution and robust AI safety research; however, as you can see from the table below, he views X Risk as a very high probability.
He puts forward his views in this TED Talk:
Will Superintelligent AI End the World?
Geoffrey Hinton, who is often referred to as the Godfather of AI, recently left Google so that he could talk freely about the threats emanating from AI. He believes that AI will do enormous good, but he also cautions that AI systems will soon be far more intelligent than humans and that there is a 15% to 20% probability that they will take over and have no need for humanity. Hinton’s 60 Minutes Interview Hinton with Andrew Marr.
We asked our various AI to summarize how the top 15 futurists view the risks in front of us. We have included their findings below.
|
Futurist |
Gen/Agentic/Robotic AI Risk (%) |
AGI Risk (%) |
ASI Risk (%) |
|
Sam Altman |
10 |
30 |
50 |
|
Elon Musk |
15 |
25 |
30 |
|
Dario Amodei |
10 |
30 |
60 |
|
Ilya Sutskever |
10 |
25 |
60 |
|
Demis Hassabis |
5 |
20 |
50 |
|
Geoff. Hinton |
10 |
20 |
25 |
|
Ray Kurzweil |
2 |
5 |
10 |
|
M. Suleyman |
10 |
25 |
45 |
|
Jensen Huang |
5 |
15 |
35 |
|
Peter Thiel |
5 |
10 |
15 |
|
P. Diamandis |
2 |
5 |
10 |
|
Mo Gawdat |
15 |
30 |
65 |
|
Yann LeCun |
1 |
3 |
5 |
|
Andrew Ng |
1 |
3 |
5 |
|
Nick Bostrom |
10 |
50 |
90 |
Another former Google employee turned concerned skeptic is Mo Gawdat, who wrote a 2021 book called Scary Smart that centers on the dual nature of artificial intelligence as both a transformative force for good and a potential threat to humanity.
Looking at the threat side of things, Gawdat asks that even if the threat is only one-in-six, why would we play the equivalent of Russian Roulette with our very existence??
He also believes AI’s advancement is inevitable and eventually beneficial but expects there will be a period of challenge before we can obtain the post-scarcity world of abundance. Gawdat also provides us with an interesting perspective and path toward that abundance in that he emphasizes that humanity’s collective behavior shapes AI’s ethical framework, and therefore as AI learns from the data and actions it observes we should make every effort to model compassion, empathy and ethical values to guide AI toward a positive future. This three- minute video from Gawdat is worth a watch.
Finally, Elon Musk agrees with Hinton’s assessment that there is a 80% to 85% probability of AI being beneficial and a 15% to 20% probability of existential risk.
Before the ChatGPT moment, no one had been more vocal than Elon Musk relative to the threats posed by AI, a threat to humanity he thought exceeded the threats emanating from nuclear war, as he unequivocally asserts in this interview.
Indeed, it was Musk’s concern relative to this threat that led him to invest $50 million in OpenAI and to call for the six-month pause, along with Max Tegmark and another 30,000 concerned citizens.
Unfortunately, that requested pause was summarily set aside and gained no traction as the tech companies blasted forward full steam ahead.
In the first 30 seconds of this video, Musk agrees with Geoff Hinton that there is a 10% to 20% risk that AI will make us extinct, but goes on to say that the positive scenario far outweighs the negative scenario.
In closing this chapter, it seems clear to us that we are living through both the most dangerous time in human history as well as a time of great opportunity. In short, we are living at the hinge of history, a concept we will explore further in the next chapter.