Why autonomous AI demands a new model of organizational authority

Mona and Luna are two “managers” in the service industry. Mona works in Stockholm, running a small café that recently opened. Luna operates a San Francisco retail shop, which turned over more than $1000 in its first four days.

Except that Mona and Luna aren’t people. They are AI agents, designed by Silicon Valley startup Andon Labs as part of a series of experiments to test whether AI can run real businesses without human oversight.

At first glance, this feels like another AI story about automation. It isn’t. Its signalling something much deeper.

AI is not just being used to support decisions anymore. It is starting to make them, and in some cases act on them. Organizations are no longer only deploying software. They are beginning to introduce systems that operate end-to-end workflows, with a degree of delegated authority.

Working with leading AI labs, including Anthropic and OpenAI, Andon Labs is building what it calls “safe autonomous organizations”. The goal is not to run profitable businesses, but to see what happens when AI systems are allowed to operate with real responsibilities, real tools and real consequences.

Built on top of models like Claude Sonnet, Mona and Luna are the full effective managers of these shops. They handle pricing, inventory, supplier coordination and even hiring personnel – from developing job descriptions to leading interviews and selecting candidates.

They are not recommending actions. They are taking them.

And, in many ways, the experiment worked. The systems were able to coordinate tasks, interact with humans and run day-to-day operations. But the failures are where things become more interesting. Mona ordered large quantities of supplies that didn’t match the menu it had designed. Luna struggled to align pricing with demand, making decisions that were internally consistent but economically flawed.

These are not just technical glitches. They point to something more fundamental. The systems can act, but they lack the implicit constraints and authorization boundaries that humans tend to apply without thinking.

And this is where the shift becomes harder to ignore. The discussion is not really about whether AI-led shops can become successful businesses. It is about the fact that systems like them are already being placed in positions where they are expected to act on behalf of an organization.

Similar signals are emerging elsewhere. Autonomous food service concepts, including fully robotic food trucks capable of preparing and serving meals, are moving from pilot to deployment. What looked experimental very recently is starting to appear in real operational settings.

AI agents are not a new phenomenon. These LLM-powered systems combine autonomy with delegation, executing tasks and making decisions with no human oversight, just observation. They have long been deployed in customer service to resolve complex queries, and in data analysis to process vast datasets and surface actionable insights. What is new is the scale and consequence of their deployment. Organizations and economies are now racing to adapt existing AI, software and risk governance frameworks to account for an operating environment they were not designed for.

Onboarding an AI agent looks surprisingly similar to hiring a person. You assess what it can do, define what it is allowed to do, outline its goals and objectives, grant it access to the systems it needs, and decide how closely to supervise it. But a human employee brings judgement, caution and skin in the game. An AI agent brings none of those things by default, which means the organization has to supply them externally through governance, structure and oversight.

Clear guidelines need to be set for AI agents, as they lack the legal accountability, moral responsibility and reputational incentives of regular employees. Consequences are not intrinsic, and governance needs to compensate for the gaps. Core concepts that need to be defined by deployers include the degree of autonomy to which an agent determines how a goal is pursued, the authority to execute and communicate across systems, and the checkpoints and boundaries to constrain it. Internal guardrails built into the system and the external controls put in place by the organization are a non-negotiable to ensure that legal standing, data security and operational integrity are never put at risk.

This is where the Agent Capability and Authorization Profile (ACAP) comes in. At its core, it is a way of documenting what an agent is actually allowed to do in a specific situation in a production-grade environment. ACAP was introduced in AI Agents in Action: A Playbook for Trusted Adoption, Authorization and Scaling, the third of a report series by the AI Global Alliance, which brings togethr leaders from industry, government, academia, and civil society to guide the responsible, ethical, and equitable advancement of generative AI.

ACAP defines, in a single record, the scope of an agent’s delegated power – detailing permitted actions, specific contexts, and required conditions – alongside assigned oversight. Ultimately, it bridges technical capability with enforceable organizational authorization.

It applies at the deployment level and is based on context, not just the technology itself. The same agent can operate under different ACAPs depending on the workflow, the level of risk, or how much autonomy the organization is comfortable granting. And when multiple agents are working together, the whole setup is treated as one system from an authorization point of view, even if it’s made up of different pieces underneath.

This stands in contrast to setups like Mona and Luna, where the objective is to observe what agents can do without human oversight and loosely defined goals. In practice, ACAP is designed for production environment, where agent behaviour must be explicitly authorized, monitored and constrained at scale.

ACAP also isn’t something you define once and forget. It evolves as the agent moves from design, to deployment, to real-world use. Early on, the boundaries might be tighter, with more checkpoints and human oversight. Over time, as the system proves itself, that scope can expand, or just as easily be pulled back if things don’t behave as expected.

In that sense, ACAP works as a bridge. It connects the technical side of what the system can do with the operational reality of what it is actually allowed to do inside an organization. And it turns what is often implicit and scattered into something explicit, trackable and, importantly, adjustable over time.

What Mona and Luna ultimately reveal is not just the emergence of more capable AI systems, but a deeper organizational shift: from systems that generate outputs to systems that exercise delegated agency.

That changes the governance question entirely.

The issue is no longer whether an AI system can assist a business. It is whether organizations are prepared to define, constrain and continuously supervise what these systems are authorized to do on their behalf.

Mona and Luna were never really experiments about capabilities of running cafés or retail stores. They were early tests of organizational delegation. The failures it outlined were not only technical failures. They were governance failures: unclear boundaries, loosely defined objectives, insufficient oversight and no operational model for authorization.

This is precisely the gap that needs to be addressed. With tools.

Rather than treating AI agents as software tools, ACAP treats them as organizational actors operating under explicitly defined authority, constraints and accountability structures. It turns delegated agency from something implicit into something governed, auditable and continuously adjustable over time.

And that distinction matters because the next generation of AI systems will not simply recommend actions. They will increasingly negotiate, coordinate, transact and operate across workflows with growing autonomy.

The real challenge ahead is therefore not whether AI can run parts of a business. In many cases, it already can.

The challenge is whether organizations, regulators and societies are ready to govern systems that increasingly act on their behalf.