Written by Jaiz Anuar

Every AI Company Wants Safety. None of Them Wants to Lose the Race.

AI safety cannot depend on good intentions alone. As capability, autonomy and adoption accelerate, organisations need security architecture that limits access, authority and consequences even when competitive pressure makes restraint difficult.

In a recent CNN interview with Anderson Cooper, former Anthropic and OpenAI researcher Jacob Coxon described why he became increasingly concerned about the direction of advanced AI development. His concern was not simply that today’s AI systems are suddenly capable of causing catastrophic harm. It was the trajectory: models are becoming more capable, more autonomous and more deeply integrated into real-world systems, while the companies building them are under intense pressure to move faster.

From a security risk architecture perspective, this is the part that matters most. The issue is not whether AI companies understand safety. Many clearly do. The issue is whether the controls, governance and risk boundaries around these systems can keep pace with the incentives pushing capability forward.

That is a very different problem.

Safety Awareness Is Not The Same As Risk Control

The major AI companies already invest heavily in safety research, model evaluation, red teaming and responsible deployment. They test for harmful outputs, cyber misuse, deceptive behaviour, biological risk and unexpected autonomy. The people working inside these organisations are often among those most aware of how quickly AI capability is changing.

But understanding a risk does not automatically mean controlling it.

Security architects deal with this problem all the time. A project team may fully understand that a system has residual risk. The business may understand that controls are incomplete. Management may recognise that a dependency is too concentrated. Yet the organisation may still decide to proceed because the commercial pressure, timeline or strategic objective is considered more important.

AI development now faces the same tension, only at a much larger scale.

The Risk Is In The Incentive Structure

A responsible AI company may decide that a model should undergo more testing before release. That sounds sensible. But what happens if a competitor is prepared to release a similar capability first?

What happens if delaying six months means losing customers, investment, talent or strategic relevance?

What happens if one country introduces strict limitations while another continues development at full speed?

This is where the risk becomes structural.

The problem is not necessarily that one company wants to behave irresponsibly. The problem is that every company can rationally argue that slowing down alone does not reduce the overall risk. It may simply reduce its own competitiveness.

From a risk architecture perspective, that is a classic control weakness: the safety decision depends too heavily on voluntary behaviour inside an environment where the incentives favour speed.

This Is Similar To Concentrated Trust

Security architecture often tries to avoid placing too much trust in a single control.

We do not assume one firewall will never fail.

We do not assume one identity provider can never be compromised.

We do not assume one administrator will always make the right decision.

Yet in AI governance, society is increasingly depending on the organisations building the technology to decide how much risk is acceptable, how much autonomy is too much and when a capability should be delayed.

That creates concentrated trust.

The organisations benefiting commercially from AI development are also being asked to determine the safety boundaries around that development.

This does not mean those organisations cannot make responsible decisions. It means the architecture of governance should not depend entirely on them doing so.

The Enterprise Version Of The Same Problem Is Already Here

The same pressure is beginning to appear inside ordinary enterprises.

Business units want AI-enabled capabilities quickly. Technology teams want productivity gains. Executives do not want competitors to move faster. Security and risk teams are expected to establish governance without becoming the reason innovation slows down.

The questions are familiar.

Can we approve this faster?

Can we launch first and strengthen controls later?

Do we really need this additional review?

Other companies are already using it.

This is where the security risk architect has an important role.

The objective is not to block AI adoption. The objective is to understand where the risk actually sits and ensure that the architecture limits the consequences when assumptions fail.

That means asking different questions.

What data can the AI access? What identity does it use? What actions can it perform? What requires human approval? What happens if the model behaves unexpectedly? Can the organisation revoke its access quickly? Can the system continue operating if the AI service becomes unavailable? Can the organisation explain and audit the decision path?

These are not AI research questions.

They are enterprise risk architecture questions.

Guardrails Are Not Enough

One of the biggest mistakes organisations can make is treating AI safety as a policy problem alone.

Policies matter.

Usage guidelines matter.

Model safety controls matter.

But if an AI system has broad access to sensitive data, privileged APIs and critical business processes, behavioural guardrails alone are insufficient.

Security architecture should assume that a guardrail may eventually fail.

An agent may misunderstand an instruction.

A model may be manipulated.

A plugin may return malicious content.

A user may unintentionally expose sensitive information.

The architecture should already have limited what happens next.

Least privilege.

Segregated identities.

Restricted data access.

Independent approval for high-impact actions.

Auditability.

Kill switches.

Recovery capability.

These controls are far more important than simply trusting the model to behave correctly.

The Real Measure Of Safety Is What Happens Under Pressure

The AI industry will continue publishing safety principles.

Enterprises will continue producing AI policies.

Governments will continue developing regulation.

The real test is not whether those documents exist.

The real test is what happens when safety conflicts with speed.

Would a company delay a model release if the safety evidence remained uncertain?

Would an enterprise postpone an AI-enabled business capability if governance was incomplete?

Would management accept slower implementation because the identity, data or approval architecture was not yet ready?

That is where risk appetite becomes real.

Security governance is easy when the control has no cost.

It becomes meaningful when someone has to accept slower delivery, reduced functionality or additional friction because the risk remains too high.

The Risk Architect’s Perspective

From a security risk architecture perspective, the solution is not to predict whether advanced AI will become dangerous in five years, ten years or never.

The job is to design for uncertainty.

Assume capabilities will continue to improve.

Assume business adoption will accelerate.

Assume AI systems will receive more data, more access and more autonomy.

Then build the controls before the dependency becomes too difficult to unwind.

The objective is not to stop the race.

It is to ensure that the organisation is not forced to choose between innovation and control after the architecture has already made that choice for them.

Final Thoughts

The most important part of the CNN discussion is not whether everyone agrees with Jacob Coxon’s view of future AI risk.

The more important point is the tension he describes between safety and competition.

AI companies may genuinely want stronger safeguards. Enterprises may genuinely want responsible adoption. Regulators may genuinely want innovation to continue safely.

But incentives matter.

When competitive pressure increases, voluntary restraint becomes harder.

That is why AI safety cannot depend entirely on intention.

It needs architecture.

It needs defined boundaries.

It needs accountability.

It needs independent challenge.

And it needs the ability to slow down when the risk is greater than the organisation is prepared to accept.

Because the real question is not whether AI companies want safety.

The question is whether the governance around AI remains strong when losing the race feels more immediate than losing control.

Question assumptions. Share knowledge. Build trust.