As we saw very clearly in Chapters Three and Four, the intelligence explosion is here. The future is here. The ChatGPT moment accelerated the arrival of AGI and ASI by two to three decades along with the risks and opportunities associated with them.
The fundamental question for humanity is, how are we going to prepare for and usher in the new future in front of us now that the timeline to prepare has been accelerated by 20 to 30 years?
What is clear to us is that as of today, the AI in place today—Perception AI, Gen AI, Agentic AI, and Robotic AI—are mostly manageable relative to existential risk and have the potential to deliver 85% to 90% of the economic gain in the near term.
However, what is also clear is that following the ChatGPT moment, AGI and ASI are about to arrive twenty to thirty years ahead of schedule. As we saw, AGI and ASI begin to represent different levels of autonomy and therefore different levels of risk. We The futurists tell us that AGI occurs when AI is as intelligent as the average human. However, as we made clear in the last chapter, in a world where AI is capable of reading 1.2 billion books a year, our AI is clearly far more capable than us in terms of processing and absorbing information. Based on these metrics one could argue that our AI is on track to become 100 million times “smarter” than us.
So, the only real change that occurs with the arrival of AGI is not an increased intelligence differential rather the fundamental change is one of autonomy. That is the change will not be that AI will all of a sudden be as smart as AI, as we saw above it is 100 million times smarter than us in narrow domains of pure knowledge. The big change will be that with the advent of AGI our AI will be able to act with agency and think strategically.
The difference will be that these extraordinarily “smart” entities will no longer need us to direct them with our prompts; they will be able to act on their own in a generalized way having gained their own agency. What they do with that mountain of knowledge and agency no one knows. Therein lies the existential risk.
In our view, once AGI happens, ASI will follow shortly thereafter because once the AGI comes into being with agency it will rapidly act to multiply adopting recursive self-improvement, and very quickly embody an intelligence that is far greater than the intelligence of all human beings put together as IJ Good predicted in 1965.
The important point to understand here is that the ChatGPT moment accelerated our reckoning with AI by two to three decades. Moreover, the reckoning has been unleashed on a disconnected, contentious, warring world, where the global systems that were put in place after our last world war are fraying if not falling apart. The ChatGPT moment has unleashed the full force of the AI reckoning on our current world order with all its strengths and weaknesses.
The ChatGPT moment has unleashed a ferocious global competition between the principal corporate actors as well as creating a high stakes geo-political competition between the leading nations of the world, particularly China and the US, as they fight for global dominance. At the same time, all of these actions put humanity on a potentially collision course with powerful AI or AGI/ASI.
The existential concern is that the extraordinary intelligence we are creating, this super intelligence might just decide to treat us no differently than we treat insects, because the intelligence differential between us and the AI will be far greater than the intelligence differential between us and ants.
If the stakes were not so high, the irony would make a wonderful setting for a Greek Tragedy. The world’s leading trillion-dollar corporations led by billionaire entrepreneurs are caught in a ferocious race to be first past the post. The billionaire entrepreneurs are not doing it for money they are doing it for fame, power and most importantly for ego.
At the same time we have the great power struggle going on between nations states with China and the US taking the lead in the fight on that front, again each determined to be first across the finish line.
These internecine competitions makes it far more likely that humanity as a whole will lose control of this process greatly increasing the chances of the existential risk emanating from AI. Rather than fighting each other we should be working together and with AI to create the world of Sustainable Super Abundance our technological genius has made available to us. Once again it seems as if we are in great need of a wisdom commensurate with our technology.
One reason the stakes in this competition are so high is the fact that most of the trillion-dollar tech companies competing for AI victory are what Eric Schmidt calls “network effect” businesses where generating a first mover advantage with AI can become a definitive competitive edge.
As Eric Schmidt explains below, for network-effect businesses, “it is the slope of your improvement that determines everything.” He goes on to explain that as AI “gets closer to superintelligence, the slope gets even steeper” and therefore if a company can crack the AGI challenge making their AI fully autonomous it can give them a competitive advantage that might never be overcome.
For the principal players engaged in this fight click here: The AI Battle for Supremacy
There is not an AI actor in the world that does not understand the risks associated with the increasing autonomy accruing to the digital species we have created. Indeed, two former Google employees Mo Gawdat and Geoff Hinton have left Google to speak without constraint on the risks facing humanity as a result of the increasingly powerful AI we are creating.
There is a broad consensus that the existential threat coming from AI taking over is somewhere between 10% to 20% or about one in six. It seems to us that Mo Gawdat is 1,000% correct when he asks why would we play Russian roulette with the future of trillions of human beings yet to be born?
In addition to the threat from existential risk, AI presents another set of challenges that humanity is ill-prepared to address given the lack of clear and consistent regulation:
- Misinformation and Manipulation
- High Unemployment and Economic Inequality
- Bias and Discrimination
- Autonomous Weapons and Battle Robots
For more information on these risks click here: AI Risks
On the other hand, to their credit, many AI entrepreneurs are begging for real, bona fide regulation. They want us to put guidelines and rules in place. They are playing a game as of now with no rules and no referees. We have never seen an industry where the principal participants are begging to be regulated. That should tell us something.
Elon Musk has been begging for increased AI regulation for years, consistently contending that AI represents far more of an existential threat to humanity than even nuclear war.
In the first minute of this short video, Elon Musk suggests that AI is perhaps the most profound existential risk that humanity has ever created and that if we are to successfully build the world of abundance now available to us we will need new strong regulation:
In fact, in an ironic twist of history, Elon Musk helped create the ChatGPT moment by contributing to the creation of Open AI, an entity which was initially set up as a not-for-profit with a mission to ensure AGI development prioritizes safety and public good over commercial interests.
In 2012, Musk met DeepMind co-founder Demis Hassabis, who warned that AI could become superintelligent and potentially threaten humanity, prompting Musk to consider the risks of a single entity controlling AGI. Sources #1
By 2013, Musk was alarmed by Google’s growing AI prowess and started a conversation with eventual Open AI co-founder Sam Altman.
In May 2015, Altman wrote to Musk, “Been thinking a lot about whether it’s possible to stop humanity from developing AI. I think the answer is almost definitely not. If it’s going to happen anyway, it seems like it would be good for someone other than Google to do it first.”
Musk replied, “Probably worth a conversation.” Six months later, OpenAI was founded as a nonprofit to counter Google’s lead, aiming to develop AGI safely and distribute its benefits broadly.
Former OpenAI researcher Daniel Kokotajlo later claimed that Musk, Altman, and co-founder Ilya Sutskever launched OpenAI to prevent DeepMind’s Hassabis from becoming an “AGI dictator,” reflecting fears of Google’s concentrated power. Musk himself stated in a 2023 Fox News interview that OpenAI was created as a “counterweight to Google,” reinforcing this narrative.
OpenAI was founded on December 11, 2015, as a nonprofit organization focused on advancing AI research for the benefit of humanity. It was established by co-founders including Sam Altman, Elon Musk, Ilya Sutskever, Greg Brockman, and others, with a mission to ensure AGI development prioritizes safety and public good over commercial interests.
In the meantime, Sir Demis Hassabis was hoping to build the “NASA of AI” following the January 26, 2014, sale of the company he co-founded with Shane Legg, and Mustafa Suleyman for $500 million. In the early days he was able to recruit many of the top AI researchers in the world as there was little competition.
According to Demis Hassabis, DeepMind’s goal is to “solve intelligence, and then use that to solve everything else,” focusing on accelerating human scientific discovery through artificial intelligence. Founded in 2010 and acquired by Google in 2014, DeepMind generated a number of successes in its quest to develop artificial general intelligence (AGI) that can learn, reason, and solve problems across diverse domains. As we have seen, they developed AlphaGo, a game that learned on its own and performed the famous Move 37 on its way to beat the global GO champion one decade ahead of schedule.
In 2020 Google DeepMind created another landmark success with AlphaFold, which solved the decades-old problem of protein folding. By predicting the 3D structures of proteins from amino acid sequences, AlphaFold achieved accuracy comparable to experimental methods, revolutionizing biology and drug discovery. Its database, released in 2021, mapped 98.5% of human proteins and over 200 million structures across species, freely accessible to researchers. This work earned DeepMind global acclaim, with applications in developing vaccines and treatments for diseases like malaria. AlphaFold’s impact underscores DeepMind’s goal of accelerating scientific progress through AI. Moreover, Demis Hassabis and John Jumper won the 2024 Nobel Prize in Chemistry for their work on AlphaFold. Mr. Hassabis was also knighted in 2024.
We mention this evolution because it seems that from afar, the Google AI DeepMind team was more focused on solving scientific discovery than they were on developing commercial use cases, at least until they could be more sure of the safety angle. In fact, rather than monetize AlphaFold, Google decided to make this program freely available to the scientific community, thereby forgoing substantial revenues if they had decided to commercialize it.
(To get a sense of the accelerating pace of change, Google DeepMind announced three days ago on June 25, 2025, the launch of a new AI, Alpha Genome, that represents another step forward in terms of improving human health outcomes: AlphaGenome).
Indeed, the mad dash that has existed since the OpenAI launch of ChatGPT was not in evidence when Max Tegmark and the Future of Life Institute (FLI) organized the first three informal AI conferences in Puerto Rico and Asilomar. In those days the field of participants was quite small and manageable. Also, the median expectation for the arrival of AGI as of the time of the first Puerto Rico Conference was 2055 as a result the discussions that took place had far more to do with the expected timing of the arrival of AGI rather than prepare for its imminent arrival.
As you can see from clicking on these documents the conversations and meetings organized by FLI seemed hopeful and on point with a sense of purpose but not urgency.
The First Informal AI Conference in Puerto Rico The Asilomar Conference Januray 2017 Puerto Rico #2 – The Beneficial AGI Conference
However, in the years following the second conference in Puerto Rico and leading up to November 2022, AI technology was dramatically improving due to advancements in transformer architecture and more powerful chips from NVIDIA. This period marked a transformative phase in AI, driven by breakthroughs in algorithms and hardware.
At the same time the escalating financial demands of scaling AI models pushed organizations like OpenAI to seek new funding strategies to remain competitive. The skyrocketing costs and competitive landscape led OpenAI, founded in 2015 as a nonprofit to develop safe AGI, to transition to a for-profit arm (OpenAI LP) in March 2019.
The nonprofit model, funded by $130 million in donations by 2019, couldn’t sustain the billions needed to train models like GPT-3 or compete with Google’s DeepMind and Microsoft’s Azure AI.
The for-profit “capped-profit” structure allowed OpenAI to secure a $1 billion investment from Microsoft in 2019, followed by $10 billion in 2023, enabling it to scale infrastructure and hire top talent.
This shift, while controversial (Musk criticized it as a betrayal in 2023 X posts), was driven by the need to fund compute-intensive research and stay in the AI race, where rivals like DeepMind (acquired by Google in 2014) leveraged massive resources.
The transition maintained a nonprofit board to oversee the mission but prioritized commercial products like ChatGPT, which achieved a $10 billion revenue run rate by 2025, reflecting the financial imperatives of the era’s AI advancements.
It was this transition that provided Open AI with the funding they needed to be in a position to launch ChatGPT in November 2022.
Frightened by the launch of ChatGPT at a time when the safety of emerging AGI was still in question, in March 2023, Max Tegmark teamed up with Elon Musk to ask for an immediate 6 month pause in the development of AI systems more powerful than GTP-4.
The Future of Life Institute (FLI) pause letter started like this:
“We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.
AI systems with human-competitive intelligence can pose profound risks to society and humanity, as shown by extensive research[1] and acknowledged by top AI labs.[2] As stated in the widely endorsed Asilomar AI Principles, Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources. Unfortunately, this level of planning and management is not happening, even though recent months have seen AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one–not even their creators–can understand, predict, or reliably control.”
We highly recommend you read the full letter here: FLI Pause Letter
The Pause letter was one last attempt to rein in an industry that was exploding at breakneck speed following the ChatGPT moment and the dramatic, accelerating advances occurring in AI technologies. In spite of being signed by a number of leading thinkers on AI, the letter was largely ignored.
The shift at Open AI from not-for-profit to for profit came to the fore in November 2023 when the OpenAI nonprofit board, fired Sam Altman as CEO and removed him from the board citing a lack of “consistent candor” in communications.
The decision was abrupt, announced via a blog post, and followed a review of Altman’s conduct. Mira Murati, the CTO, was named interim CEO. The board’s statement suggested Altman’s actions undermined their ability to oversee OpenAI’s mission, though specifics were initially vague.
ThefiringoccurredamidtensionsoverOpenAI’srapidcommercialization, with ChatGPT’s success and Microsoft’s $10 billion investment in 2023 shifting focus toward profit. The board, tasked with prioritizing the nonprofit mission, reportedly clashed with Altman’s aggressive push for growth.
Eventually, they explained that he was let go because of lack of transparency, mission drift, internal conflicts and personal ambition. However, Altman was quickly reinstated a few days later when the vast majority of Open AI employees signed a letter demanding his reinstatement.
It is worth noting that the OpenAI employees held equity in the for- profit arm which was valued at $86–$90 billion in 2023, per Forbes.
Altman’s leadership drove this valuation through ChatGPT’s success (100 million users by January 2023) and Microsoft’s $10 billion investment. His firing threatened a potential collapse in OpenAI’s market position, as competitors like Google ($2 trillion market cap) and Anthropic ($61.5 billion) could capitalize on the chaos. A mass resignation to join Microsoft, as threatened in the letter, offered employees a chance to preserve or even enhance their equity value, as Microsoft proposed hiring them under Altman to form a new AI research unit.
Elon Musk has filed multiple lawsuits against OpenAI and its CEO Sam Altman, primarily alleging that the company breached its founding mission to develop artificial intelligence (AI) for the benefit of humanity by prioritizing profits through its for-profit arm and partnership with Microsoft.
Whatever the results of the lawsuit, one thing is clear the move from not-for-profit to for-profit enabled Open AI to partner with Microsoft to secure the funding they needed to build out ChatGPT and be the first company to launch Generative AI in November 2022, marking the next stage in human technological history and perhaps human history itself.
The race for AGI is now led by a cohort of trillion-dollar companies committed to winning the race, seemingly with little or no thought to cost or impact. If the decision were ours to make we would pause the development of AGI for several years so that we could be sure to manage the risks going forward. That being said, it seems very clear to us that is highly unlikely a pause can be engineered.
Indeed, Max Tegmark has said famously that there were two guiding principles in thinking about how to contain AI’s autonomous development: don’t provide AI with access to the internet, and don’t teach it how to code. In the race for power, both are long gone!!
That being said, the former UK Prime Minister took the initiative to call for the first-ever government-led summit in Bletchley Park in the UK in November 2023 and Max Tegmark has continued to fight for AI safety.
That UK-led coalition-building initiative has been followed by several additional government-led coalition-building initiatives where the Future of Life Institute turned advisor rather than organizer. As you will see below when you review the summits that have been organized and conducted this is still an ad hoc process where neither the US or Chinese governments have taken the lead but it has produced some important guiding documents as you will see below.
Bletchley Park, UK: November 1-2, 2023
Co-chaired by UK Prime Minister Rishi Sunak and South Korean President Yoon Suk Yeol, this was the first government-sponsored conference of its type. It was a landmark global conference organized by Sunak to address the safety and regulation of frontier AI—highly advanced general-purpose AI systems. Hosted at Bletchley Park, the historic World War II codebreaking site symbolizing technological innovation, the summit convened representatives from 28 countries, including the United States, China, and the European Union, alongside tech leaders including Elon Musk and Sam Altman (OpenAI), academics such as Yoshua Bengio, and civil society groups.
Initiated by the UK’s Department for Science, Innovation and Technology (DSIT) under Michelle Donelan, the event aimed to foster international cooperation to mitigate AI risks, including terrorism, disinformation, and existential threats from uncontrolled systems. The summit resulted in the Bletchley Declaration, a commitment to safe, human-centric AI, the establishment of the UK’s AI Safety Institute, and a mandate for an International Scientific Report on Advanced AI Safety led by Bengio, with follow-up summits planned in South Korea (2024) and France (2025). You can read the declaration here: The Bletchley Declaration
You can read PM Sunak’s Summary Here: UK PM Rishi Sunak Summary of the Conference
AI Seoul Summit; May 21-22, 2024
Co-chaired by UK Prime Minister Sunak and South Korean President Yoon Suk Yeol, the UK and Korea agreed to continue to work together with the AI Seoul Summit, held on May 21–22, 2024. This was a hybrid event comprising both virtual and in-person components. The Leaders’ Session on May 21, co-chaired by UK Prime Minister Rishi Sunak and South Korean President Yoon Suk Yeol, was conducted virtually, allowing heads of state and global leaders to participate remotely.
The Digital Ministers’ Meeting on May 22, co-hosted by the UK’s Department for Science, Innovation and Technology (DSIT) and South Korea’s Ministry of Science and ICT, took place in Seoul, involving ministers, industry representatives (e.g., from Google, OpenAI), and academics. Additionally, the AI Global Forum, hosted by South Korea’s Ministry of Foreign Affairs, was an in-person event on May 22, broadening participation. The virtual format of the Leaders’ Session, often referred to as a “mini virtual summit,” was designed to maintain momentum from the Bletchley Park Summit (2023) but was criticized for limiting engagement compared to fully in-person summits like Bletchley or Paris (2025).
Here are the principal declarations to come out of Seoul: Seoul AI Summit
Paris AI Action Summit February 10-11, 2025
Co-chaired by French President Emmanuel Macron and Indian Prime Minister Narendra Modi, the next major international AI summit after the AI Seoul Summit was the AI Action Summit, held on February 10–11, 2025, at the Grand Palais in Paris, France. The summit marked a shift from previous gatherings by emphasizing actionable initiatives and global inclusivity in AI development.
Conference summary and declarations here.
The Paris AI Action Summit faced significant criticism from the AI safety community. Experts expressed disappointment that the summit prioritized economic growth and innovation over addressing the pressing safety risks associated with advanced AI systems.
Notably, the United States and the United Kingdom did not sign the summit’s final declaration, which emphasized “inclusive and sustainable AI.” The U.K. cited concerns that the declaration lacked practical clarity on global governance and did not sufficiently address national security issues. The U.S., represented by Vice President JD Vance, warned that excessive regulation could stifle innovation, suggesting that the summit’s approach might hinder technological advancement.
Max Tegmark, the MIT physicist and president of the Future of Life Institute, was among the prominent voices expressing concern over the summit’s outcomes. He described the event as a “real bellyflop.” He went on to say, “It almost felt like they were trying to undo Bletchley.” Tegmark emphasized that the summit failed to build upon the safety commitments made at previous gatherings, such as the Bletchley Park Summit, which had focused on mitigating AI risks. He warned that neglecting AI safety in favor of rapid innovation could lead to uncontrolled developments with potentially catastrophic consequences. Recap here
This was disappointing because it seemed as if momentum for reasonable regulation had been building following Bletchley Park and Seoul. Indeed, those previous conferences had commissioned the first- ever formal AI Safety Report, which was prepared by Yoshua Bengio and delivered to the conference. You can find a link to the report here: First AI Safety Report
The Singapore Conference
The Singapore Conference on AI: International Scientific Exchange was a one-day conference organized on April 26, 2025, in response to the perceived shortcomings of the Paris AI Action Summit. The Paris Summit, co-chaired by France and India, was criticized by AI safety experts (e.g., Dario Amodei and Max Tegmark) for de-emphasizing safety in favor of economic opportunities and inclusivity, with the US and UK refusing to sign the Statement on Inclusive and Sustainable AI due to its lack of practical clarity on governance and safety. This fragmentation, coupled with the summit’s shift away from the safety focus of the Bletchley Park (2023) and Seoul (2024) summits, prompted Singapore to refocus global efforts on technical AI safety research.
The Singapore government, leveraging its role as a neutral tech hub, acted swiftly after Paris to address the “diplomatic setback” and safety concerns, as noted in reports, with the conference’s outcomes published in the Singapore Consensus on May 7, 2025. While Singapore’s AI governance initiatives (e.g., the AI Safety Red Teaming Challenge, November 2024) predate Paris, the April 26th conference was a direct attempt to salvage the damage done by Paris’s diluted safety agenda and bridge East-West divides before the next summit in India in November 2025.
In his remarks, Tegmark stated, “After its near-death experience in Paris, international AI safety collaboration is roaring back.”
It seems as if the conference successfully got AI Safety back on the top of the agenda with this document being published on May 7th: The Singapore Consensus on Global AI Safety Research Priorities
For more on the Singapore Conference: Singapore Declarations and Summary
The US Approach to AI Regulation
On January 23, 2025, President Donald Trump signed Executive Order 14179, titled “Removing Barriers to American Leadership in Artificial Intelligence.” This directive marked a significant shift in U.S. AI policy, emphasizing deregulation and rapid innovation to bolster America’s global AI dominance. The key objectives of the policy were to revoke the previous administration’s policy and request that the new AI Czar, David Sacks, put in place an AI strategy by July 22, 2025.
In sum, Executive Order 14179 underscores the Trump administration’s commitment to reasserting American leadership in AI by fostering an environment conducive to innovation and reducing regulatory barriers. While this approach aims to enhance economic and national security interests, it also raises questions about the balance between rapid technological advancement and the need for ethical oversight.
The United States played a significant supporting role in the Singapore Conference and the Singapore Consensus, contributing through researchers (Tegmark, MIT, Stanford), industry leaders (OpenAI, Anthropic), and the U.S. AI Safety Institute’s collaboration with Singapore’s Digital Trust Centre.
These efforts shaped the Consensus’s technical framework, focusing on risk assessment, trustworthy systems, and post-deployment control, building on the International AI Safety Report. However, the U.S. government, under the Trump administration and David Sacks, did not lead or formally endorse the Consensus, reflecting EO 14179’s deregulatory stance and the U.S.’s withdrawal from Paris’s cooperative framework. Singapore’s neutral mediation enabled U.S.- China dialogue, but U.S. influence was limited by its unilateral policies, setting a complex stage for the India AI Summit (November 2025).
The India Summit Scheduled in November 2025
The India AI Summit is the next major AI conference scheduled, expected to occur between November 2025 and January 2026, (most likely November 2025 based on announcements at the Paris AI Action Summit). To be hosted by Indian Prime Minister Modi and organized by the Ministry of Electronics and Information Technology (MeitY).
Status of Governmental AI Regulation
Before conducting this research, we had not fully understood that so much work has been done in terms of attempting to regulate AI, both from a civic society vantage point, like the Future of Life Institute, and from governmental action.
The Trump administration and its new AI Czar, David Sacks, plan to come up with a new AI strategy for the U.S. by the end of July. We await the final definition of the new strategy once Mr. Sacks’s report is prepared. It isn’t clear whether the India Summit will follow in the footsteps of Singapore or Paris.
Geo-Political Concerns
It’s apparent that China does not like that the US has put restrictions on the kinds of chips that can be sold to China, as the former CEO of Google, Eric Schmidt, stated in this TED Talk, which we highly recommend.
Schmidt also highlights the geopolitical risks associated with any near- term breakthrough toward AGI. After explaining the Chinese annoyance with the export restrictions, he describes a frightening scenario:
“Let’s assume I am the bad guy, and you are the good guy. You’re six months ahead of me and we are both on the same path to superintelligence. And you are going to get there, right? And I am sure you are going to get there. You are that close. And I’m six months behind. Pretty good, right? Sounds pretty good. No!! These are network-effect businesses. And in network-effect businesses, it is the slope of your improvement that determines everything. So, I’ll use OpenAI or Gemini, they have 1,000 programmers. They are in the process of creating a million software AI programmers. What does that do? First, you don’t have to feed it except for electricity. So that’s good. And they don’t quit and things like that. Second, the slope is like this. Well, as we get closer to superintelligence, the slope gets even steeper. If you get their first, you dastardly person, I will not be able to catch you. And I have given you the tools to reinvent the world and, in particular, to destroy me. That is how my brain Mr. Evil is going to think. So, what is the www am I going to do?
“The first thing I am going to try to do is to steal all of your code. You have prevented that because you are good. Second, I am going to infiltrate you with humans. Well, you have good protections against that as well. You don’t have spies. So what do I do? I am going to go in and change your model. I am going to modify it. I am going to screw you up to get me, so I am one day ahead of you. And you’re so good I can’t do that. What’s my next choice? Bomb your data center.”
Existential and Geo-Political Concerns
In chapter 3, we referred to the Economist interview of Sir Demis Hassabis (born 1976), the head of Google AI, and Dario Amodei (born 1983), the founder of Anthropic, where we shared their views on the timing of AGI, here we want to share with you their concerns and foreboding as it relates to the lack of regulation of AI.
Lord Hassibis explains that civil society does not fully understand how fundamentally AGI/ASI will impact our world. He hopes that as people more fully understand how positive these changes can be, a sort of CERN for AGI will be created where the scientific world collaborates on the last few steps to achieve AGI by generating global norms. For more on CERN.
Dario Amodei then explains why Paris was a missed opportunity:
“We are on the eve of something that has great challenges, it is going to greatly upend the balance of power. If someone dropped a new country into the world, ten million people smarter than any human alive today, you would ask them the question, what is their intent? What are they actually going to do in the world, particularly if they are able to act autonomously. I think it is unfortunate that there wasn’t more discussion of those issues as there was in the early summit that the UK held, the one in 2023 in Bletchley Park. I hope that future summits reclaim this mantle.”
When asked to comment on potential over-regulation in Europe and under-regulation in the US, Lord Hassabis suggests that the right answer is likely somewhere in the middle, but always being sure to balance against the two types of major risks he sees, one being bad actors who use these powerful technologies for nefarious means and:
“The other being from agentic systems themselves getting out of control or not having the right values or the right goals. Both of those things are critical to get right, and I think the whole world needs to focus on that.”
They are asked on a personal level, do they ever worry about ending up like Robert Oppenheimer?
Lord Hassabis did not hesitate to respond:
“I worry about those kinds of scenarios all the time. That is why I don’t sleep very much. I mean ,there is a huge amount of responsibility on the people, probably too much, leading to this technology. I think that is why we and others are advocating that we probably need new institutions to be built to help govern some of this. You know I talked about CERN, I think we need a kind of equivalent IAEA Atomic Agency to monitor sensible projects and those that are more risk-taking. I think that society needs to think about what kind of governing bodies are needed. Ideally, it would be something like the UN, but given the geo-political complexities, that does not seem very possible. So, I worry about that all the time. We just try to do everything we can in the vicinity and influence that we have.”
Dario agreed with his contemporary’s comments:
“My thoughts exactly echo Demis. It feels like every decision I make is balanced on the edge of a knife.”
These leaders are calling for our global civic society to help them think through and manage this unprecedented moment in history. That is the purpose of this book: to generate a wider global civic society discussion around these momentous technologies and their potential for positive and negative impact on the world.
In closing this chapter, we are going to leave the reader with two excellent TED Talks from important actors who believe we should effectively pause the development of AGI.
We highly recommend that readers watch these videos, where the founder of the Future of Life Institute at MIT, Max Tegmark, and technologist Tristan Harris both make the case for holding back on the development of AGI until we can be more certain our interests are aligned with the digital species we are creating.
How to Keep AI Under Control | Max Tegmark | TED
Why AI Is Our Ultimate Test and Greatest Invitation | Tristan Harris | TED