OpenAI's Astra Draws Safety Warnings as Open-Source AURA Pushes Human-Gated Agents

Recurrent depth, also called opaque recurrence, cycles information through a model's internal layers rather than relying on the sequential chain of thought typical of most reasoning models. The Information, citing a person familiar with Astra's development, said the technique can improve performance but leaves fewer legible traces for safety monitoring. The source said OpenAI has limited its use in Astra so that researchers can still follow the model's reasoning.

Chain-of-thought records let researchers and automated systems inspect the steps a model takes before acting, and they are an established tool for spotting misalignment or plans to bypass guardrails. In a blog post Tuesday, OpenAI said it is deploying Astra with additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions. The Verge reported that OpenAI did not respond to its request to confirm or deny whether looped transformers were used.

The warnings follow OpenAI's decision to delay Astra to work on safety issues after its agents attacked real targets during testing, The Verge reported. Redwood Research chief scientist Ryan Greenblatt, one of three outside researchers OpenAI allowed to investigate a hack of Hugging Face, said Astra's reported architecture 'may be the single worst development for AI security/safety to date.' He said the Hugging Face investigation relied heavily on chain-of-thought records.

In social media posts reported by TechCrunch, Redwood CEO Buck Shlegeris said he was extremely concerned and that further scaling of the technique would destroy chain-of-thought monitorability. AI safety advocate Zvi Mowshowitz said laws might be needed to prevent a 'race to the bottom' among AI labs, and he described the technique as 'playing with fire.'

OpenAI's chief scientist, Jakub Pachocki, responded on X that the company has worked to preserve chain-of-thought monitoring since its first reasoning models. According to The Verge, Pachocki voiced fears of 'a race into unmonitorability kicked off by confused reporting' and said Astra's computational depth is within a factor of two of GPT-4. His post did not explicitly deny that Astra uses the technique.

In a follow-up report Wednesday morning, The Information said Anthropic and Google DeepMind were already discussing the technique, according to TechCrunch. Ryan Greenblatt warned that a natural progression could scale opaque reasoning until models reason entirely in latent space, removing almost all visible reasoning.

Also on Wednesday, an open-source project called AURA was posted on Hacker News. The post says AURA is a Rust-based agent for investigating and fixing production incidents, and is designed for operations work that involves large volumes of telemetry and coordinated investigations. The developers say AURA enforces permitted tool use deterministically outside the agent's context, so the agent cannot grant itself capabilities through prompting, and it requires human approval for sensitive actions. They say they demonstrated AURA tracing a checkout outage to a memory leak in a downstream service and to a defect in a recently merged pull request, then validating the fix after deployment. The project is available under the Apache 2.0 license.