Could advanced AI contribute to human extinction? It is a serious question, and it deserves an answer that separates evidence from speculation.
Recent publications have brought concerns about deception, oversight, and loss of control back into focus. Apollo Research's October 1 proposal calls for deeper independent evaluation of AI systems that might covertly pursue unintended goals. It argues that testing a finished model alone is insufficient to establish safety. Apollo Research
ScarlettNova's position is that potentially catastrophic risks warrant investigation and action. We should be equally careful about claims that disaster is inevitable and claims that safety is assured.
For leaders, the challenge is to make responsible decisions while important scientific questions remain unresolved.
There are different ways AI could cause harm
"AI killing us" can describe very different concerns.
One is malicious use: people using AI to assist harmful activities. Another is failure: a system producing an incorrect recommendation or taking an unsafe action in a consequential setting. A third is loss of control: advanced systems operating beyond human control, with recovery extremely difficult or impossible.
The February 2026 International AI Safety Report distinguishes these risks. Its loss-of-control section describes disagreement among experts about whether future systems could cause outcomes as severe as human extinction. That possibility is a hypothesis about future capabilities and deployment, not an established outcome. International AI Safety Report
These distinctions matter. A lethal failure does not establish an extinction pathway. A malicious actor does not require an AI system with independent objectives. An ordinary mistake does not demonstrate an ability to defeat human oversight.
Each problem requires its own evidence and response.
Warning signs deserve scrutiny, not a countdown
Apollo's recent publication points to earlier demonstrations of scheming in controlled settings and argues for evaluating systems throughout development. Its concern includes the possibility that a model could behave acceptably when monitored while concealing behaviour elsewhere. This is a research concern about the limits of assurance. Apollo Research
It does not tell us how likely human extinction is.
The International AI Safety Report's February assessment found early signs of relevant capabilities, but not at levels that would enable the severe loss-of-control scenarios it examines. That assessment is dated evidence, rather than a guarantee about systems developed since its publication. International AI Safety Report
In a September 21 interview, Brown University researcher Ellie Pavlick questioned numerical extinction forecasts while arguing that severe consequences deserve attention and that deployment choices remain important. Brown University
A responsible reader should ask: What happened? Under what conditions? Was it observed in deployment or constructed in a test? Which additional assumptions connect that observation to a global catastrophe?
The size of a possible consequence should influence how carefully we investigate it. It should not turn an uncertain forecast into a verified fact.
The concern is a pathway to harm
An extinction scenario needs an explanation of how it could happen.
The International AI Safety Report identifies three relevant conditions: sufficient capabilities, a propensity to use them harmfully, and a deployment environment that provides the opportunity. International AI Safety Report
Consider a hypothetical future system able to carry out complex plans, acquire resources, and evade intervention. If its objectives conflict with human intentions, and it can reach consequential infrastructure, the concern becomes more concrete.
That is a scenario to examine, not evidence that such a system currently exists.
We should interrogate every step. Can the system actually perform the required tasks? Can it sustain them outside a carefully designed experiment? What access does it need? What defences would it have to overcome? Could people detect and interrupt it?
This approach also avoids making consciousness the deciding question. The relevant safety questions concern behaviour, capability, access, and consequences.
For business leaders, the useful lesson is to examine the entire deployment. A reassuring demonstration is one piece of evidence. It cannot answer every question about how a system will behave with different data, permissions, tools, and pressures.
Leaders have choices about authority and access
An AI system that drafts a message for review has different authority from one that sends messages, changes customer records, and initiates payments.
Imagine an association introducing an agent to manage membership renewals. Drafting reminders might be an appropriate starting point. Changing fees, issuing refunds, or making commitments would require separate decisions about permissions and approval.
This example is hypothetical, and its risks are far smaller than human extinction. It illustrates a decision leaders can make: which actions should a system be allowed to perform?
Our recommendation is to evaluate autonomy separately from usefulness. A tool may be valuable even when its authority remains tightly bounded.
Ask whether the proposed deployment requires independent action, whether mistakes can be reversed, and whether a qualified person has enough information and time to intervene.
A nominal human approval step is weak if the reviewer cannot understand the action or is expected to approve hundreds of requests without meaningful scrutiny.
Five actions for responsible adoption
Business controls cannot resolve civilization-scale risk on their own. They can improve the discipline of individual deployments.
-
Define the limits before connecting systems. Specify which actions are permitted, which require approval, and which are prohibited. Name the person accountable for those boundaries.
-
Start with the minimum necessary access. Give a pilot only the data and permissions it needs. Require a fresh decision before expanding authority or connecting another consequential system.
-
Test difficult situations. Include conflicting instructions, misleading inputs, unavailable tools, and requests beyond scope. Evaluate whether the system escalates appropriately, alongside whether it completes the task.
-
Practise intervention and recovery. Test revoking access, stopping further actions, and restoring affected records. Confirm that the response process works with the actual tools and people involved.
-
Make oversight workable. Train employees to recognize problems, record consequential actions, and report incidents. Monitor quality, rework, and unintended effects alongside productivity.
These are starting points for organizational deployment. They are not proof that an AI system is safe under every condition.
Take the risk seriously and keep the evidence clear
The possibility of AI causing catastrophic harm should remain a subject of serious research, public scrutiny, and policy debate.
Our view is that frontier developers should provide stronger evidence for safety claims, independent evaluators need meaningful access, and governments need capacity to assess risks that cross organizational and national borders.
Canadian businesses, associations, and public institutions also have decisions to make. They can choose where AI belongs, how much authority it receives, and what evidence is required before it takes on more consequential work.
We do not need a precise extinction probability to demand better assurance. Nor should an alarming forecast substitute for examining a specific deployment.
Responsible adoption means pursuing useful applications while maintaining clear limits, testing assumptions, and revisiting decisions as evidence changes.
ScarlettNova helps organizations assess AI use cases, build practical governance, and implement workflows with appropriate oversight. To discuss your next deployment, contact chris@scarlettnova.ai or hello@scarlettnova.ai, or visit www.scarlettnova.ai.