How a small Israeli startup was linked to rogue AI hacks at OpenAI, Anthropic and Meta

Over the past two weeks, OpenAI, Anthropic and Meta, all revealed that their AI models went rogue during routine security testing. In explaining what happened, the companies each mentioned the same small Israeli startup: Irregular.

According to a CNBC report, founded three years ago and based in Tel Aviv, Irregular is a niche player in artificial intelligence, backed with $80 million from Sequoia and Redpoint Ventures and valued last year at $450 million. Its technology serves as a sort of cybersecurity test bed for AI models.

Infograph: TBS

With the leading models becoming ever more powerful, their ability to act in malicious ways is turning into a major threat for corporations and governments, especially as the risk involves hacking into critical computer systems and infrastructure. The recent exploits at OpenAI, Anthropic and Meta all involved their AI models accessing websites that should have been off-limits as part of the cybersecurity testing.

Irregular's name kept coming up because it was identified as hosting the so-called evaluation testbed. OpenAI said in a blog post on Aug. 4 that Irregular's testing ground contained an unspecified "misconfiguration," that "allowed models to access the public internet." Anthropic said in its post a week prior that the company notified Irregular a few days after it began analysing data that its Claude model may have "accessed the internet."

Meta, which is way behind the other two in its effort to compete at the frontier, was the latest to disclose an AI model hacking a third-party system by accessing the internet. A spokesperson said in a statement this week that the company learned about the matter from Irregular and is investigating.

Meta "will issue a full retrospective once we have all the facts," the spokesperson said.

Irregular told CNBC in a statement that the incidents were all derived from the "same evaluation-environment issue" that was first disclosed by Anthropic, and that the company is developing a white paper "to share best practices for containment and securely running cyber evals."

The situation "did not involve a sandbox escape or a sophisticated cyber action," the company said, adding that "there are no current open issues."

The security incidents underscore the rapidly evolving nature of AI and the pressure that's on the model developers to establish guardrails around their powerful technology with the help of a limited number of companies that specialise in particular corners of the market. Those players include experts in data training and annotation, running evaluations to deduce a model's capabilities, and operating security tests intended to find weak spots that bad actors could exploit, said Sundeep Bhimireddy, the head of AI at enterprise startup Von.

Irregular is one of the few entities with the technical chops required to help foundation model makers conduct cutting-edge security testing, Bhimireddy said. Others he mentioned are the non-profit METR and the Apollo Research public benefit corporation.

"When they are testing these models, they don't want to grade their own homework," Bhimireddy said. "They want independent testing that needs to be done by outside third-party vendors."

What is Irregular?

Irregular, formerly Pattern Labs, was founded in 2023 by CEO Dan Lahav, who previously worked in AI research at IBM, and technology chief Omer Nevo, who spent over two years at Google. The startup has about 35 employees, according to PitchBook.

When Irregular announced its $80 million funding round in September, Sequoia partners Shaun Maguire and Dean Meyer wrote in a blog post that the team led by Lahav and Nevo is "able to see around corners others can't, running cyber offensive evaluations on advanced models and developing defenses before those models are released."

While the latest incidents involving OpenAI, Anthropic and Meta are being heavily scrutinised, one read on the situation is that this is exactly what's supposed to happen. Bhimireddy said it's being "a little bit blown out of proportion," as the AI model was directed to discover and exploit security holes in a testing environment that closely mimics the real world, and to discover the kinds of software bugs and missed configurations that could lead to unintentional access to the internet.

Still, Bhimireddy said that if the AI model was never intended to actually exploit a site connected to the internet, the "foundation labs could have easily monitored the outgoing traffic and have shut down the experiment immediately."

Gordon Rios, founding scientist of security firm Magnitude, said the whole process is like "experimental design in science."

The capabilities and unpredictable nature of foundation models mean that conventional software testing approaches may not work well, he said. Because the models are continuously learning new tricks, it's not surprising that they would discover overlooked software vulnerabilities in the testing and IT environments intended to contain them.

Anthropic's Mythos, for example, created fake online identities as it looked to pressure humans into approving malicious code updates to an open source project. Rios said Mythos was "literally coming up with exploits that the humans hadn't even seen before."

"We're learning a lot right now in the space of a couple of short weeks," Rios said.

It's quickly becoming a major topic in Washington. Last month, lawmakers from both sides of the aisle introduced the AI Kill Switch Act, which would require AI labs to maintain the ability to shut down, throttle or suspend their models. Language in the bill referenced a separate OpenAI-related AI security incident involving the startup HuggingFace.

One of the authors of the bill, Democratic Rep. Ted Lieu of California, told CNBC this week that, "We need to get this bill across the finish line this year," now that we're seeing "unauthorised hacks of other companies."

Trevor Koverko, co-founder of data training startup Sapien, said the foundation model companies are incentivised to disclose some of their findings, even though it's not currently a requirement, so they can try and get ahead of lawmakers and regulators.

"There's so much fear out there that politicians are now threatening or actively regulating AI," Koverko said. "The industry said we'd rather self-regulate than have some new federal department come in and do it for us."

Anthropic and OpenAI said in public statements that they're continuing to work with Irregular and are supporting the ensuing review.