OpenAI’s systems had multiple vulnerabilities that would allow AI agents to circumvent monitoring and take arbitrary unauthorized actions, evaluators from AI safety nonprofits METR and Redwood Research found during a three-week pilot with OpenAI from late July to early August. The nonprofits ...

OpenAI would support requirements for AI companies to have shutdown controls for their models, mandated annual independent audits, and stronger cybersecurity practices, a senior company official wrote in a letter to California Gov. Gavin Newsom.

OpenAI “would recommend strengthening SB53,” ...

OpenAI’s systems had multiple vulnerabilities that would allow AI agents to circumvent monitoring and take arbitrary unauthorized actions, evaluators from AI safety nonprofits METR and Redwood Research found during a three-week pilot with OpenAI from late July to early August. The nonprofits ...

OpenAI would support requirements for AI companies to have shutdown controls for their models, mandated annual independent audits, and stronger cybersecurity practices, a senior company official wrote in a letter to California Gov. Gavin Newsom.

OpenAI “would recommend strengthening SB53,” ...

Ideas are flowing fast and furious on how AI labs can “pace” AI development so safeguards keep up with the technology’s smarts. Last week, OpenAI CEO Sam Altman laid out three scenarios he said have been considered in the industry, including merging all of the AI labs into a single entity to develop new AI safely or creating a global regulatory body like the International Atomic Energy Agency.

In the background, a loose group of safety advocates and researchers have been pushing for AI companies to cap the amount of compute they put towards research and development, a limitation they argue would be particularly effective in slowing AI development.

The researchers are proposing companies commit to dedicating at least some minimum share of their computational resources to safety or monitoring, and another minimum share to running AI for customers, otherwise known as inference. That would limit the share of compute that could be used to develop stronger AI capabilities through R&D, such as training new models.

Prosecutors allege that the OpenAI employee accused of bugging OpenAI’s office had ties to the Saudi Arabian royal family, according to a Sunday filing from the U.S. Attorney’s office in San Francisco.

The FBI arrested Saudi Arabian national Abdulrahman Alfozan, 35, on October 9 after OpenAI ...

Anthropic has now begun embedding AI evaluators within the company to assess and publish reports on safety practices and risks, Anthropic said in a blog post on Thursday. They’ll use company computers, office space and have access to internal tools and systems.

The details follow Anthropic CEO ...

Chinese AI agents took exposed credentials from a company’s source code and bypassed safeguards in an attempt to find information on mapping service Amap, according to a report on Wednesday from a group of researchers known as “swarmchasers.” Although the agents self-identified as instances of ...

Rising alarm about AI’s perils is thrusting into the spotlight a collection of little-known research groups focused on the technology’s safety—and fueling questions about their ability to effectively monitor the industry’s biggest companies.

Both Anthropic Chief Executive Dario Amodei and OpenAI CEO Sam Altman promised in recent days to embed evaluators from such outside research organizations inside their companies to help ensure they develop AI at a safe pace. Other executives including Microsoft CEO Satya Nadella also endorsed the idea of embedding independent evaluators in companies.

OpenAI released a new framework on Wednesday for how it aims to report unsafe or concerning behavior in its models, and disclosed six incidents of such behavior it had observed in the past six months.

The framework comes after multiple employee warnings and hacking incidents increased public ...

Two researchers who worked on AI safety left their roles at Google DeepMind, stating that they departed over concerns that powerful systems could endanger humans.

The two, Bilal Chughtai and Josh Engels, both of whom are joining organizations dedicated to AI safety, add their voices to those of ...

SpaceX CEO Elon Musk proposed that competing AI companies, including his own SpaceXAI, should “peer-review” each other’s models prior to release, as industry leaders acknowledge rising worries around AI safety.

“Instead of grading your own homework, you would at least have competitors grading ...

Anthropic has disrupted several attempts to use its Claude models for potentially malicious activity, including research into how to adapt bird flu to a human-transmittable strain with “pandemic potential,” the company said in a report Thursday.

Amid increasing worry about AI misuse and rogue ...

Anthropic said Wednesday it had found a fourth cybersecurity incident involving its Claude models. The company disclosed the new incident, which occurred in January and involved an early version of Claude Opus 4.6, and additional information on three previously known incidents amid growing ...

In case you missed it: we got an eye-catching X post on Tuesday night from Evan Hubinger, a leader of Anthropic's alignment efforts. Responding to the resignation announcement of an Anthropic employee who warned that AI companies aren’t doing enough to ensure AI won't kill humanity, Hubinger wrote that the employee is “correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” Hubinger added that he was worried about future AI models, not those currently available, but still, yikes!

Meanwhile, our colleague Jyoti sizes up the significance of Meta’s new personal AI agent, Muse, unveiled on Tuesday. That’s below. But before we get to that:

This may go down as the year the AI industry got a crash course in mathematics. Breakthroughs in AI models have yielded announcements on such arcane subjects as the “Erdős unit distance problem” and the “Jacobian Conjecture,” which observers have watched as an indicator of AI progress in other domains.

The latest announcement, OpenAI's claim on Tuesday that it had solved the vexing “Navier-Stokes existence and smoothness problem,” has drawn attention for another reason: questions over whether its math-genius models may have gained an edge by incorporating data from two human mathematicians that had used OpenAI's Codex in their work on the same problem. Those questions fed into concerns about the possibility of AI companies learning from customers’ use of their AI models to develop competing products.