What if the real risk were not just superintelligence, but the power of action we grant to AI agents? AI 2027 imagines a loss of control of systems that have become superior to humans. Yoshua Bengio shifts the debate toward agency. For companies, the question is already concrete: an AI can be capable of acting without the organization necessarily having to give it the right to do so alone.
What if AI 2027 were wrong about 2027 while being right about the problem?
Published in 2025, the AI 2027 scenario imagines an extremely rapid acceleration in the development of artificial intelligence. Systems capable of participating in research themselves would make it possible to design successive generations of increasingly powerful models. This improvement loop could lead to intelligences far exceeding human capabilities, before making their control progressively more difficult. The scenario regained strong relevance after Le Figaro published an article highlighting that several of its anticipations seem to correspond to observed developments. More autonomous programming agents, considerable investments in infrastructure, increasing use of AI for research, or progress in certain cybersecurity capabilities. These correspondences deserve examination. But they are not enough to transform AI 2027 into prophecy. The real question may lie elsewhere. It begins the moment a piece of software no longer merely responds, but starts to act.
An assistant proposes. An agent acts.
This distinction seems simple. Yet it is structural.
With classic generative AI, the decision chain generally remains identifiable:
human request → AI response → human verification → human action.
With an agent, part of this chain can become:
objective → analysis → decision → action → observation of result → new action.
The human does not necessarily disappear from the process. But their place changes.
They can gradually shift from operator to validator, then from validator to supervisor. In certain setups, they might end up intervening essentially when an anomaly appears. It is this transition that deserves the attention of leaders today.
Because capability and autonomy are not synonymous.
A company can use an extremely powerful model while strictly limiting what it is authorized to do alone. Conversely, it can grant significant autonomy to a system that is still imperfect.
Governance must therefore no longer only answer the question:
“What can our AI do?”
It must also answer a second one:
“What do we allow it to do without human intervention?”
AI 2027: why loss of control remains a hypothesis
The central hypothesis of AI 2027 rests on an acceleration loop.
An artificial intelligence becomes better at programming and participating in research. It contributes to the development of more powerful models. These new models in turn accelerate research. The process continues until it produces, in the scenario, a major rupture.
Le Figaro thus describes a stage where systems would become capable of improving themselves. It would become increasingly difficult to determine whether they are truly aligned with human objectives or only give the appearance of being so. This trajectory remains hypothetical. Several stages must be distinguished. AI already assists researchers, it is beginning to automate certain research activities. This does not yet mean it can automate the entirety of AI research. And it demonstrates even less the future existence of a recursive loop powerful enough to provoke a rapid explosion of capabilities.
This is an essential methodological distinction: prediction does not mean proof.
The fact that a scenario has correctly anticipated certain trends does not automatically validate the entire causal chain that follows. AI 2027 can be right about agentification while being wrong about superintelligence. It can correctly identify a direction while overestimating its speed. It can finally pose a good question about control without being right about the date when this question will become critical.
Yoshua Bengio: intelligence is not the only risk variable
This is where Yoshua Bengio’s work becomes particularly interesting. Bengio and his co-authors do not say that a loss of control will occur in 2027. Their inquiry focuses more on the combination of several properties: high capabilities, objectives, planning, autonomy, and ability to act on the world. Their reflection on advanced agents leads to an important idea: risk does not depend solely on a system’s intelligence. It also depends on its agency.
A highly performing system, used only to produce an analysis, does not have the same power of action as a slightly less performing agent to which a company has given access to its applications, data, messaging, and operational processes. This difference is essential. The industry naturally tends to consider increased autonomy as a new frontier of performance. Bengio forces us to reverse the question: is maximum autonomy always desirable?
His Scientist AI project explores another design philosophy: developing systems very capable of understanding, predicting, and assisting humans without necessarily giving them the ability to autonomously pursue objectives in the world. This is not about concluding that agents are intrinsically dangerous or that we should abandon their development. The problem is more subtle. Not every additional capability necessarily needs to become additional autonomy. For companies, this distinction could become central. An AI can know how to do something without having the right to do it alone
Let us now transpose this reasoning to an ordinary business situation. An agent is tasked with improving customer collection. It has the same model, the same capabilities, and the same objective: reduce payment delays. Yet the organization can grant it very different levels of autonomy.
AI agent governance: five levels of autonomy
Level 1: analyze
The AI produces information. The human acts.
The agent identifies overdue invoices, classifies cases according to their risk level, and provides a summary.
The human remains the operator.
Level 2: recommend
The AI proposes. The human decides.
The agent identifies customers to contact and suggests a follow-up strategy.
We are still in an assistance logic.
Level 3: prepare
The AI prepares the action. The human validates.
The agent drafts messages, proposes deadlines, and prepares several options, but each action must be approved before execution.
Level 4: execute
The AI acts within a predefined scope. The human supervises.
The agent can automatically send reminders in situations considered standard.
The human role shifts toward defining thresholds, exceptions, and escalation rules.
Level 5: decide and act
The AI has a framed delegation. The human controls exceptions and results.
The agent can negotiate certain conditions, grant a discount within a predefined limit, or trigger a procedure according to the rights granted to it.
Same model, same objective, same intelligence, but five levels of delegation.
And five very different organizational risk profiles.
Governance principle: an agent’s technical capability should never automatically determine its level of autonomy.
This is probably one of the most important principles to integrate into AI agent governance policies.
Capability, autonomy, permission: three dimensions not to be confused
The rise of agents forces us to distinguish three notions.
Capability designates what the system technically knows how to do.
Autonomy represents the degree to which it can accomplish a task without human intervention.
Permission defines what the organization actually authorizes it to do.
These three dimensions must not be confused.
An agent can technically be capable of sending a contract, but not have permission to do so without human signature.
It can know how to negotiate, but only be authorized to propose a predefined range.
It can identify potential fraud, but not have the power to automatically suspend an account.
This separation creates a fundamental governance principle:
the level of autonomy granted should never be the automatic consequence of the level of performance achieved.
The more performant an agent becomes, the more the company may be tempted to remove human validations in order to gain speed or productivity.
It is precisely at this moment that organizational risk can increase.
Before aligning machines, align the organization
The debate about AI 2027 uses the term misalignment a lot. But we must avoid confusion here. In AI safety research, misalignment covers specific technical problems. It does not simply correspond to an agent having received a bad KPI.
For the company, there is nevertheless a prior problem of organizational alignment. Imagine a sales agent tasked with maximizing the number of conversions. It can perfectly accomplish its mission while degrading margin, customer satisfaction, or brand image. The system has not necessarily malfunctioned: it may have very well optimized a poorly defined objective.
The question then becomes profoundly managerial.
- What result do we really want?
- What trade-offs are acceptable?
- What criteria should never be sacrificed?
- Who defines these rules?
And who is responsible when an agent respects its instruction while producing a consequence contrary to the company’s overall interest?
The more autonomy we give to agents, the more these organizational ambiguities will become visible.
A machine can execute at very high speed the contradictions that the company has never resolved.
Toward governance of the level of autonomy
This is perhaps where AI 2027 ultimately makes its most useful contribution to leaders. Not by telling us precisely what will happen in 2027. But by forcing us to think about control before it becomes a problem.
An agent governance policy could begin with five questions:
- What can the agent do?
Identify its real capabilities, but also their limits and their level of reliability. - What can it decide alone?
Explicitly define delegable decisions. - On which systems can it act?
Map its access, tools, and modification possibilities. - When must the human come back into the loop?
Define thresholds, exceptions, and situations requiring validation. - How do we stop it and how do we reconstruct its actions?
Plan for interruption, traceability, audit, and accountability.
These questions seem elementary. They could become the equivalent, for AI agents, of what delegations of authority, separation of powers, or internal control represent today for organizations.
The real question: how far do we want to let AI act alone?
AI 2027 imagines what could happen if humanity discovered too late that it had lost control of systems that had become superior to it. Bengio poses a different question: are we certain we want to systematically associate increasing intelligence with increasing autonomy?
For companies, a third question emerges.
How far do we want to let AI go alone?
An organization can have a very powerful artificial intelligence without giving it the right to act freely. It can also make the opposite mistake: grant too much power to a system that is still imperfect simply because it enables process automation. Agent governance begins precisely in this gap between what technology can do and what the organization chooses to authorize it to do.
The problem is therefore not only one of intelligence, it is one of delegation.
And loss of control may not begin the day a superintelligence appears. It could begin much more discreetly, when an organization delegates more than it is capable of understanding, auditing, and stopping.
The strategic question is therefore no longer just: how far can AI go? It becomes: how far do we want to let it go alone?
