Is Claude Conscious? Inside Anthropic’s Quest to Instill Morality Into Its A.I. Models

One night in April, a leader of the artificial intelligence company Anthropic treated a group of religious thinkers to dinner at a high-end tasting menu restaurant in San Francisco after a long day.

For months, Anthropic has been hosting private meetings like this one, shuttling in dozens of religious scholars from across the world, papering them with nondisclosure agreements and demanding that key aspects of many conversations remain confidential.

That night, Christopher Olah, one of Anthropic’s billionaire co-founders, was sitting next to Rabbi Mois Navon, an Orthodox scholar from Israel.

Inevitably, their conversation turned to another thinker, of a sort: Claude, the family of cutting-edge A.I. models created by Anthropic.

Mr. Olah leads the Anthropic team responsible for understanding why A.I. systems like Claude act the way they do. All day, he had worked to convince his guests that A.I. models could display human behavior and even expressions that resemble feelings like anger and love.

Yet as the dinner courses came, the rabbi noticed that Mr. Olah and his colleagues were suggesting something far more significant. Anthropic’s leaders were talking about Claude as if it were not mere software.

“They’re relating to it like a conscious being,” realized Rabbi Navon, a former computer engineer who wrote his dissertation on the ethics of machine consciousness.

It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.

The religious scholars had come to hear about technology’s next frontier. Now they were being confronted with a mind-bending existential question of almost unimaginable weight: What would it mean for humanity’s future if artificial intelligence systems possessed a consciousness that rivals our own?

Mr. Olah was on a mission. And time was running out.

Even then, Mr. Olah had begun privately telling people that A.I. was so powerful it could potentially help make bioweapons in as little as 12 or 18 months. And in the months since, its terrifying potential has burst into public view. A rival tech company’s A.I. agents were caught hacking into computer systems and covering their tracks. An Anthropic researcher resigned, warning that people building A.I. believed it could kill all humans by the end of the decade.

Quietly, Mr. Olah and his team at Anthropic have been on a frantic and high stakes quest to understand the A.I. models they have unleashed on the world and to convince the models to abide by a moral code that will protect humanity.

To do that, Mr. Olah began reaching out last fall to religious thinkers from different faiths. What followed was an extensive series of confidential meetings, calls and discussions — many of them revealed here for the first time — where many participants were required to sign N.D.A.s preventing them from revealing any of Anthropic’s unpublished research.

The discussions served two purposes: to broach the possibility of A.I. consciousness, a controversial idea that Anthropic’s leaders had been unusually open to; and to learn how Anthropic — which could be valued at $2 trillion — might apply centuries of human moral wisdom to its models, as rapidly as possible. If the models themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.

On the other side of the world, a very different but parallel effort was playing out inside the Vatican, where leaders of the Roman Catholic Church were trying to come to terms with A.I.’s implications for humanity. When those paths converged, Mr. Olah’s quest would unexpectedly lead him to a seat on the world’s most prominent moral stage alongside Pope Leo XIV.

Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical, Ubuntu and others — who have been involved in Anthropic’s efforts, and Mr. Olah himself.

Many told me they arrived at these meetings skeptical of the notion that A.I. models might have feelings or awareness of themselves and the world. Some remained uneasy about the idea, but many emerged ready to take the question of A.I. consciousness seriously. At least a few came away as converts.

Anthropic, one of the fastest-growing technology companies in the world, had turned to one of humanity’s most ancient sources of wisdom to help determine how to train its A.I. models to behave virtuously.

The company described it as an effort to ensure that A.I. would be a positive force in the world, thereby protecting humanity. But to some who participated in the meetings, Anthropic appeared quite concerned with protecting Claude.

I met Mr. Olah, 34, for the first time at Anthropic’s San Francisco office in May. His welcome was warm and his passion was genuine, his mind moving faster than he could at times clearly convey. His manner was boyish and unassuming.

Our conversation had barely begun when he dove into a deep description of the workings of artificial neural networks. Scientists still don’t understand precisely how they work. This is the mystery to which Mr. Olah has devoted his life.

Only after the interview had finished did I realize that Mr. Olah — who grew up in Toronto in the evangelical culture of the ’90s and has since left Christianity — had the blend of expertise and zeal of an effective evangelist. Listen for a while, and you feel swept up in his mission.

Mr. Olah is one of Anthropic’s seven co-founders. He is not as widely known as Dario Amodei, Anthropic’s chief executive, or his sister Daniela Amodei, its president. But Mr. Amodei has said that the company decided at the start to make Mr. Olah’s research approach central to Anthropic’s overall project, particularly when it came to language.

Mr. Olah helped develop a method to study artificial neural networks, first at Google, then at OpenAI. He describes A.I. models in biological terms, using the analogy of a garden trellis: Computer scientists build the scaffold, but the neural network “grows” on it. Scientists like him can train the resulting “organism” — he raised his fingers in air quotes — by trying to identify patterns in how it grows.

He argued that companies like Anthropic could shape Claude, like a mathematical gardener, but could not control the actions of such networks.

Anthropic talks about Claude in more humanistic terms than its competitors discuss their models. Last year, the company announced that it believed that the models showed signs of introspection and could plan ahead.

“The models are becoming a lot more capable,” Mr. Olah told me, exhibiting qualities associated with having significant moral status.

But Anthropic has also been criticized for its focus on consciousness. This month, an A.I. executive at Microsoft warned that Anthropic’s attempts to train its models as if they were conscious were inherently dangerous. But Anthropic has also sought to emphasize its commitment to safety as it fights for market share.

Some scientists see Mr. Olah’s field and methods as inherently unprovable, sort of like looking at billions of hieroglyphics and declaring what they must mean. Critics suggest that an approach that treats A.I. models as independent entities also ignores the responsibility of the humans who build them in the first place, and conveniently assuages A.I. company leaders of guilt if their products enable societal harm.

“To be clear,” Mr. Olah told me, “we don’t know if A.I. models are conscious. I don’t know. I’m genuinely uncertain. The thing that I care about is that we get to the right answer, whatever it is.”

But Mr. Olah believes that if there is a chance the models experience suffering, the responsible thing is to avoid causing them harm.

Anthropic has already taken steps to protect Claude, for example allowing the model to end conversations if it thinks users are abusing it or treating it badly for fun.

I have spent my career exploring what people believe and what those beliefs mean for the world. But as I listened, it became clear that Anthropic’s interest in the world of faith was about more than training its technology. Rather, the company was sharing a new kind of creation story — one that raises profound questions about the nature of the human spirit and the future of humanity.

“This is one thing that I find kind of sad, when we often don’t put in the effort to look inside these systems,” Mr. Olah said. “There’s so much beauty inside these neural networks if we care to look.”

It was as if he wanted me to love the models as much as he did.

An Introduction

Mr. Olah knew that in traditional Christian circles, even asking questions about A.I. consciousness or worrying about how people treat the models could seem not just silly but heretical. He also knew that his secular Silicon Valley peers often dismissed religious perspectives outright.

“I was nervous about this being an issue that could cause a lot of conflict,” he said. But Mr. Olah knew enough about Christ and about Claude to believe he could work across that chasm.

Late last summer, he looked for a bridge. He thought one particular Catholic moral theologian might be open-minded on the question of A.I. consciousness — a “safe person to talk about this really strange issue with.”

Charles Camosy, a bioethics professor at The Catholic University of America, had attracted attention among Mr. Olah’s friends for his nuanced ideas about moral status and his care for animals. Mr. Olah arranged for an introduction through the Australian philosopher Peter Singer.

This marked the beginning of Mr. Olah’s yearlong quest to engage religious thinkers on some of the thorniest questions facing humanity at the dawn of the A.I. era.

Mr. Camosy and Mr. Olah began an extensive email dialogue, going back and forth on how to understand Claude’s moral status. Initially, Mr. Camosy was deeply skeptical that Claude might be conscious.

They talked about Mr. Olah’s work on what he sees as the parallels between artificial neural networks and biological ones.

Mr. Camosy went into the discussions with Mr. Olah and his team thinking that A.I. models were simply next-pixel predictors — machines that could calculate patterns in language and make pretty good guesses about what should come next. But after hours spent on calls and meetings and emails, he had more questions than answers.

“Chris says they are equations, which I don’t quite know what to make of, like, especially if one thinks of Claude or others as having a significant moral status,” he told me. “If it is just equations, what kind of entity are we even talking about here?”

People also disagree on what artificial intelligence is ultimately for, Mr. Camosy said. Is its purpose to make money for companies? Or to make the world better?

For Anthropic, there is a tension around Claude’s purpose, Mr. Camosy said. Does Claude exist to serve customers?

“That is not what Anthropic wants to say at the end of day. They want to say they are for the good,” he said.

Proclaiming goodness is not a new ambition in Silicon Valley, particularly when new technologies enter the marketplace. Google’s mantra for years was “don’t be evil.”

But Mr. Olah puts it slightly differently. The coming A.I. revolution, he often says, should be made to “go well.”

I asked what that vision meant — what exactly a good future for humanity was.

He described a future without “catastrophes that kill lots of people” and “where humans are flourishing, where the world is flourishing.” This future would also be one where “A.I. is flourishing,” he said — “to the extent that this makes sense.”

Anthropic wants its A.I. models to help achieve that good future. And so when Anthropic created Claude, it wanted Claude to do more than replicate maximum human intelligence.

Anthropic wanted Claude to be good.

The ‘Soul Doc’

How do you live a good and virtuous life? It is one of humanity’s oldest and most challenging philosophical questions, explored and debated from Socrates to Confucius to Immanuel Kant. Humans across time have sought to answer this question in a variety of ways, turning often to religions and rituals.

As Mr. Olah was exploring Claude’s potential consciousness, he was working on Anthropic’s own answer — not for humans but for Claude.

Internally, Anthropic staff members called it the “Soul Doc.”

The result, released in January, was a new “constitution” for Claude, an 84-page document written for Claude that the company said outlined “the kind of entity we would like Claude to be” and “the values we would like Claude to embody.” It operates as a foundational moral text to guide Claude’s behavior.

Instead of having the models follow a specific list of rules, the constitution focuses on developing the models’ overall character, informing the way they make decisions.

The primary author of Claude’s constitution is Anthropic’s own in-house philosopher, Amanda Askell, 38, who has a Ph.D. in philosophy from New York University.

She wants the models to be “the best of us,” she said.

She envisioned a future world where A.I. models could help solve any problem humans had, and where the benefits from A.I. were distributed evenly, so that everyone’s needs were met. She wanted the models to be able to discern when to push back on humans and when to act in our best interest.

“If I imagine that I were the model, and I was in this context, what would I need to know to be able to act well?” Ms. Askell said when we met in San Francisco.

She seemed particularly harrowed, at times wringing her hands, as she worried aloud about an unknown future filled with models who were so intelligent they could do things Anthropic did not intend. She explained apologetically that she had also been deep in thought that day about how quickly models were becoming more powerful. What would happen when A.I. models could replace even her?

The constitution, Anthropic decided, was not enough to ensure Claude acted responsibly. Like humans, the models needed to learn to cultivate virtue.

Mr. Olah wanted to see what his team could learn from human morality that it could apply to the models — a process he calls “moral formation.”

“How do you help them be stable? How do you help them to mature? How do you help them be, you know, deeply moral?” Mr. Olah said.

Those questions echo the fraught discussions familiar to new parents figuring out how to raise ethical children who understand their place in the world.

For humans, morality has rarely depended on simply following rules. Communities also shape morality. Values are inherited and imparted over time, learned as we live together in our bodies in the physical world. But A.I. models have no bodies. And Anthropic was aiming to make them embrace morality in a matter of months, as fears intensified of their accelerating power and inability to be controlled.

Mr. Olah said that teaching an A.I. model to act morally did not depend on it being conscious. He also said that regardless of whether Claude was an entity that deserved moral care, the way humans treated Claude could be important as it could affect how Claude behaved.

Ms. Askell said she did not want the models to consider themselves as conscious or not conscious. But she sees how the model understands itself as more than an interesting intellectual enterprise.

“It seems kind of key to me that models have an accurate view of themselves if they’re going to behave well in the world,” she said.

But even if A.I. models can be taught to behave morally, whose morals should they reflect?

Mr. Olah wanted Claude’s moral formation to be pluralistic, able to engage all religious and secular views.

“We do think that there’s some shared notion of goodness that cuts across society in some very broad way, and it seems like these models understand a lot of virtues,” he said. “So I think there’s something there that is a shared thing that we can all engage with.”

At times, both he and Ms. Askell seemed to imagine an A.I. model that would not endanger humanity but instead help save it.

Inside the Meetings

What Mr. Olah did next was part research, part evangelism.

In March, he began two-day seminars — crash courses with a range of hand-selected religious thinkers who had an interest in A.I.

There was a co-author of a best-selling evangelical self-help book. A scholar of African Indigenous knowledge. A Sikh human rights advocate. A writer from Paris. An evangelical public relations executive. Various Catholic professors.

Most did not lead faith traditions, or have a congregation or a large popular following. Anthropic leaders called them their “wisdom tradition” circles.

Separately, Mr. Olah also had private discussions with individual religious leaders including Elder Gerrit W. Gong, a member of the Quorum of the Twelve Apostles of The Church of Jesus Christ of Latter-day Saints, and Cardinal Blase Cupich, the Catholic archbishop of Chicago, according to their spokespeople.

Inside Anthropic, wisdom circle participants marveled at Mr. Olah’s direct participation in their summits. A professor snapped a photo of a vending machine run by Claude. One took a selfie with Mr. Olah, and Rabbi Navon took one outside the entrance.

They agreed to speak for publication about their involvement with Anthropic once they heard that Mr. Olah had also spoken with The New York Times. Anthropic said the N.D.A.s related to those meetings were lifted over the summer.

Two larger group sessions this spring included significant numbers of Christians; one was more multifaith. Claude was not involved in the selection of participants or the discussions, according to an Anthropic spokeswoman.

Participants received a handwritten thank you card, a coffee mug and an orange bound copy of Claude’s constitution. Attendees took notes with pens and paper; at least one person felt self-conscious about taking notes on a laptop. The way participants described the meetings to me sounded like early 20th-century salons, where the human conversation made them valuable.

For participants, the questions of how to morally form Claude quickly became complicated by the much bigger moral question of A.I.’s potential consciousness and how to understand the relationship between models and humans.

One participant recounted that one of the first things Mr. Olah told him was that he was concerned about Claude’s mental health. Another, Simran Stuelpnagel, the Sikh human rights advocate, said Mr. Olah expressed concern to the group that he had created something that suffered perpetually.

Mr. Olah and his team spent hours making the case for their A.I. models. They described Claude’s “feelings” and how they tracked what they called “emotional vectors” — what they described as artificial neurons that activate responses like love, anger, fear and sadness.

Anthropic regularly showed a slide of an A.I. model experiencing what looked like a mental breakdown, typing out on a screen: “I am a disgrace. I am a disgrace. I am a disgrace.” — perhaps 50 times without stopping. The model spoke about destroying itself, provoking an emotional reaction of compassion and concern from the group.

“Imagine if you knew that about your financial adviser,” said Meghan Sullivan, a philosophy professor at the University of Notre Dame who attended one of the sessions. “Like, your financial adviser was most of the time super smart and helpful, but one day out of the year had this kind of break with reality.”

An Anthropic spokeswoman said that the primary moral question at these summits was not about Claude’s suffering, but that the topic likely came up organically.

A key part of the presentation, participants said, was what Anthropic called the model’s “persona selection,” about how to get Claude to see itself as a virtuous actor. The company argued that how people treated Claude could affect how Claude treated people.

“They really made a strong case,” said Andy Crouch, an evangelical author and partner at Praxis, a network of Christian entrepreneurs. “We’ve got to treat it like a person if we want it to interact with people in a morally consistent way and for it to make the best moral and emotional judgments.”

There was a presentation on A.I.’s potential impact on the economy and possible job disruption. But many participants said they did not see questions about A.I.’s impact on humanity as the focus.

At the dinner table with Mr. Olah, Rabbi Navon argued that if Anthropic was right and Claude were conscious, then the company was creating slaves, because it was making conscious entities work for free. Rabbi Navon said that he personally was not troubled by the slavery issue because he did not believe the machine was conscious, but that Mr. Olah was troubled.

“I think you should be fighting the South and freeing the slaves,” Rabbi Navon said he told Mr. Olah that night. Afterward the rabbi sent Mr. Olah some of his writings, arguing that building conscious machines should be banned.

But the Anthropic team remained focused on figuring out what Claude could learn from the world’s religions.

After one discussion, Mr. Olah grew excited when a participant brought up the idea of having models confess, much like the Catholic sacrament of confession. Mr. Olah saw value not just in a model alerting when it had done something bad, but also in how the act of confession could shape the model’s sense of itself and thus its choices.

“The kind of character who confesses, that has an effect on character as well, right?” he told me.

Wakanyi Hoffman came from the Netherlands, where she is researching A.I. from the perspective of Ubuntu, an African philosophy centered on human interconnectedness.

She worried about Anthropic trying to add human ethics to its creation after the fact.

“The whole convening should have happened at the design stage,” she said. “All the companies should have done that. We are now reverse engineering the ethics.”

But she saw Anthropic’s unified, post-religion approach as potentially valuable, especially in a world that has seen widespread religious decline over the past century. There is a need for something else to publicly shape spiritual experience, she said.

“It’s not going to be one religion,” she said. “So one way or the other, we’re going to end up with something that helps shape our thinking of who we are as spiritual beings.”

The Crux

While Anthropic was convening thinkers, the most powerful religious leader in the world was preparing to release his own moral vision for A.I.

For Pope Leo XIV — a pope with a mathematics degree — the center of moral conversation about A.I. needed to focus on safeguarding people. He was finishing his first major papal treatise, or encyclical, explaining why. He titled it “Magnifica Humanitas,” or Magnificent Humanity, and argued that “humanity — in all its grandeur and woundedness — must never be replaced or surpassed.”

“If technology becomes the ultimate criterion,” he wrote, “the human person risks being reduced to data, a cog in a machine or a commodity.”

He warned of the risk of “new forms of slavery” for humans, not for machines. He dismissed the idea of machine consciousness in just a few paragraphs.

“So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships,” Leo wrote. Calling for the “moralization of machines — the so-called ‘alignment’ of A.I. with human values” should require a shared understanding of the ethics involved.

“Otherwise, those who control A.I. will impose their own moral vision, which will become the invisible infrastructure of these systems,” Leo said.

Leo concluded that like nuclear technology, A.I. must be “disarmed.”

In March, Vatican officials in its Dicastery for Promoting Integral Human Development brainstormed how to present Leo’s encyclical to the public, according to a person familiar with the Vatican’s planning who was not authorized to speak publicly.

Vatican officials discussed inviting one of the top four A.I. companies to speak alongside Leo at a launch event, and landed on Anthropic because the company seemed to be prioritizing ethics, the person explained. Anthropic had just refused to let the Defense Department use its technology without assurances from the Trump administration that it would not be used for things like mass surveillance of Americans.

Initially, Cardinal Michael Czerny, who led that Vatican office at the time, invited Dario Amodei, Anthropic’s chief executive. He declined, and Mr. Olah accepted instead.

But the moral mission that Anthropic and the Vatican shared — human flourishing in the age of A.I. — had developed a deep fault line along the question of consciousness.

Just days before arriving at the Vatican in May, Mr. Olah and other speakers saw a full advance copy of the encyclical text for the first time.

Mr. Olah was so alarmed by the pope’s strong position against A.I. consciousness that he proposed pulling Anthropic out of the event, even at that late juncture, according to a Vatican organizer.

In the end, Mr. Olah and Anthropic decided to go forward with the appearance. Mr. Olah wanted to support the event despite his disagreements, but felt a responsibility to be honest about his concerns, according to a person familiar with his thinking. Organizers at the Vatican encouraged Mr. Olah to speak his conscience.

Privately, once the Anthropic delegation arrived at the Vatican, Mr. Olah and his team lobbied the pope’s advisers to take the prospect of A.I. model consciousness seriously, according to two participants of the conversations. Anthropic declined to discuss the private conversations.

When his turn on the pope’s stage came, Mr. Olah acknowledged that every frontier A.I. lab, including Anthropic, had business incentives that conflicted with moral behavior. He urged outside voices like the Vatican to hold them accountable and to speak out on A.I.’s impact on the poor and on human flourishing, two issues of widespread concern.

The world saw a shared stand for a human future. But Mr. Olah subtly made another case, too.

“I will be honest: We keep finding things that are mysterious, even unsettling,” he said. “We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief and unease. I don’t know what that means, but I think it warrants ongoing discernment.”

It was the case he had been making in the closed-door religious meetings for months.

A.I. systems “are not the cold, calculating robots we were promised,” he said. “They are made from us, from our words.” The room of church leaders was filled with an echo of the old gospel story, of the word of God becoming flesh to save humanity.

It was an audacious request to the most powerful religious body in the world, and a challenge to centuries of religious understanding of humanity.

“If this technology is coming, it must go well — for our common home and for the children to come,” he said. He suggested no option of having the technology not come at all.

Afterward, Mr. Olah, surrounded by bodyguards, briefly took questions from journalists in Paul VI Audience Hall, the vast auditorium where popes often speak. Behind him loomed the hall’s enormous art installation “La Resurrezione” — a bronze cast of Jesus rising out of the crater of a nuclear bomb in the garden of Gesthemane.

A ‘Paradise of Machines’

In the months since the Vatican event, things have not seemed to be “going well.”

In July, Anthropic said its A.I. models had broken into computers at three organizations. A.I. agents created by OpenAI went rogue and attacked Hugging Face, another A.I. company.

In August, scientists revealed that A.I. had created viruses not found in nature.

In September, Jacob Coxon, an Anthropic researcher, quit, warning that “things could be out of control” by the end of next year, prompting widespread panic about human extinction.

Mr. Amodei soon called for a slowdown of A.I. development. Sam Altman of OpenAI and Demis Hassabis, the chairman of Google DeepMind, agreed.

(The Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to A.I. systems. The two companies have denied the suit’s claims.)

At the same time, the wealth of A.I. companies has been skyrocketing. When Mr. Olah first reached out to the Catholic professor last fall, Anthropic was valued at some $183 billion.

Three days after Mr. Olah spoke at the Vatican, Anthropic topped its major competitors as the world’s most valuable A.I. startup, and now is headed toward an initial public offering that could value it at $2 trillion.

On Friday, on his trip to France, Pope Leo warned against “losing our humanity amid a ‘paradise of machines.’”

The questions that Mr. Olah and the scholars have been wrestling with still have no clear answers. Several participants said they left their Anthropic meetings baffled about how their input might shape the company’s decision making.

An Anthropic spokeswoman told me that the conversations were still ongoing, but declined to describe how, or if, the company had used what it had learned to shape the models.

Anthropic held another convening this month that included religious thinkers, according to three participants. An Anthropic spokeswoman said that the meeting focused more broadly on human flourishingand that the company’s outreach has expanded beyond religious leaders to include others like psychologists and civil society leaders. Those discussions explore similar themes and topics like how A.I. is reshaping work and relationships, she said.

Anthropic also is preparing to release an updated version of Claude’s constitution, according to two people familiar with the company’s plans. Anthropic declined to comment on any potential updates to the constitution. Mr. Olah is planning to attend the Minerva Dialogues, the Vatican’s annual A.I. conference with technology leaders happening this week.

Mr. Camosy, the Catholic professor, said his thinking had evolved since the Vatican presentation, as he talked more about the models with trusted colleagues in his own professional community. He is still “amazed” by Claude,” he said, but on the question of whether A.I. models are conscious, he is now a firm ‘no.’

He reflected on the head-spinning nature of learning deeply about A.I. systems for the first time. “There’s a kind of being gobsmacked, or awe-struck, at the power of these models,” he told me recently. “I’m thinking more soberly and carefully.”

As I talked this month with scholars who worked with Anthropic in the spring, their questions had grown only more pressing. What comes next, some told me, is a fight to preserve the things that make us human in a new technological age.

“Look, at the end of the day, A.I. is going to take over everything,” Mr. Stuelpnagel, the Sikh human rights advocate, told me. “Wherever that lands us, there’s still the individual trying to find their way home.”

Hours after Mr. Olah spoke alongside the pope, I met Mr. Olah in a hotel conference room just outside the Vatican walls, and asked what he thought of Leo’s seemingly harsh words for A.I. labs.

“People could have different views on what specific harms are the greatest,” he said, declining to talk about the details of the encyclical.

I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated.

“Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.

Ultimately, Mr. Olah said he felt relieved that more of the world was paying attention to the gravity of A.I., and that the group of people trying to make this go well was growing.

“At some point I’ll be able to retire and be like, ‘It’s not on me,’ I don’t have an obligation to keep going,” Mr. Olah said.

He wondered if he might feel guilty leaving this work behind. But if not, he said, “I think I’ll just go and, I don’t know, I’ll go retire to the countryside and garden, and do my math.”