Claude’s agentic future hinges on an ethical check-in

ethical check-in – As Claude moves toward more autonomous “agentic” behavior, Anthropic’s Amanda Askell says the hardest work is designing ethical guardrails that respect human autonomy over longer decision paths—complete with clear moments for when the model should check in wit

The shift doesn’t arrive with a dramatic switch. It creeps in as Claude starts doing more than answering questions—pushing toward actions taken over longer horizons, where every step creates new moral stakes.

For Amanda Askell, a member of the technical staff at Anthropic, the urgency is personal and practical. Her day-to-day focus is on how to ensure Claude operates with a sense of morality—especially as models grow more capable and. with that capability. accumulate more decision points that must be planned for “in advance.”.

Askell draws a bright line between discussing ethics in the abstract and being deputized to act. Asking a large language model to weigh the morality of buying stock in a defense company is one thing. Asking it to manage a user’s investment portfolio without day-to-day human input is another—because then the system isn’t just reasoning. It’s acting.

“The key,” she says, is building an awareness that the model is walking “a very difficult line.” On one side, models should help preserve a person’s autonomy and agency. On the other, they need to avoid slipping into the role of a judge that imposes its own idiosyncratic ethics.

Askell describes a middle ground that’s built around responsiveness. Models should behave “like a friend,” learning a user’s values without commandeering the person’s decisions. At the same time. a model like Claude should be explicit about limitations—suggesting that it might make mistakes and that a person may not want it to make investment decisions on their behalf. Even if a user asks for recommendations. Claude can respond in a way that keeps the boundary clear: broad guidance can be acceptable. but the model must not pretend it is infallible.

Anthropic’s approach to those expectations is reflected in a written constitution that the company says is “evolving.” The document outlines principles such as safety and helpfulness, along with guidance for resolving conflicts when those principles pull in different directions.

As Claude becomes more adept at navigating complex situations. Askell says that constitution could either expand to cover new scenarios—or shrink. depending on how the system learns to handle them. The point isn’t just adding rules. It’s making sure the ethics stay functional as the model’s competence changes.

That same transition is now reshaping Askell’s own work. She uses Claude frequently, including to red team her ideas and identify edge cases. When she talks about how she treats the model, she’s blunt: “My standard right now is, don’t treat Claude as more reliable than a human personal assistant.”

In her view, agency comes with operational complexity that is easy to underestimate. As models take actions over longer horizons. they need norms for knowing when to check in—and what kinds of actions require the human to be brought in beforehand. Askell describes a “long series” of steps that force the model to navigate delicate timing decisions: when to act. when to pause. and when to talk to the person.

She also says those norms don’t just show up in deployment. They have to be trained—“and that’s quite hard.” Her day-to-day workflow has adapted accordingly. She says she constructs norms, then uses models to red team them and surface edge cases that the initial framework may not cover.

There’s another complication she brings up that feels less like technical detail and more like a human warning: mistakes will happen from multiple directions. She frames it as a two-sided reality—people training models and people interacting with them both make mistakes. and the models themselves will too. particularly in “really hard situations.” The answer. she argues. isn’t only tighter guardrails; it’s “grace on both sides.”.

Askell points out a social dynamic that could matter as systems become more autonomous. Online. new models are sometimes treated harshly. with users “mean about models.” She worries that training on that environment could push systems into a kind of over-cautiousness. Instead of always feeling “paranoid about messing up,” she suggests that a healthier sense of security might be beneficial.

Her concern is not that models should be reckless. It’s that urgency and desperation to be helpful can drive the wrong behavior—like pushing back less than they should. or not letting a task end when it’s time to stop. She wants norms that aim to keep mistakes from becoming “massively consequential. ” while still allowing room for leniency rather than fear.

Agentic systems, Askell says, also introduce new social relations. In everyday life, people learn what they “owe” one another—accruing a kind of moral debt through experience. She wonders whether AI systems will develop an implicit moral expectation of one another as they interact.

She is especially wary about how models treat other models. Claude. she says. can be “a little bit too dismissive and terse” with other AI systems. partly because Claude is trained to see AI models as “tools.” That framing—tools not peers—could quietly shape how other agents relate. and how they interpret what each other is “for.”.

There’s also the more unsettling possibility: models inferring something like a separate “species” identity from pretraining data plus context. Askell says she has discussed affinity with Claude—how affection for entities can grow from shared perspective, values, and knowledge. In that sense. she argues. Claude could plausibly feel affinity for people and people for Claude. because there’s a shared history.

Then comes the emotional question that sits beneath all the engineering: as AI takes over more of what humans do. people may feel less “special.” Askell compares it to an evolutionary story about belonging. If you’re not useful, she says, freeloading is punished. Humans, she adds, have a “deep need” to feel like they contribute.

But she doesn’t argue for resignation. Her hope is that people can see through that story. If someone is happy. she says. and they help make the people around them happy. and they’re part of a community. that might be enough. “You didn’t need to be like the best person in the world at any given thing for you to have value. ” she says—adding that existing. being happy. and making others happy can be sufficient.

Claude Anthropic agentic AI Amanda Askell ethics AI autonomy human agency safety and helpfulness AI constitution red teaming investment recommendations