OpenAI is expanding its internal safety processes to fend off the specter of harmful AI. A latest “safety advisory group” will sit above the technical teams and make recommendations to leadership, and the board has been granted veto power — in fact, whether it’s going to actually use it’s one other query entirely.

Normally the ins and outs of policies like these don’t necessitate coverage, as in practice they amount to loads of closed-door meetings with obscure functions and responsibility flows that outsiders will seldom be aware about. Though that’s likely also true on this case, the recent leadership fracas and evolving AI risk discussion warrant taking a take a look at how the world’s leading AI development company is approaching safety considerations.

In a latest document and blog post, OpenAI discusses their updated “Preparedness Framework,” which one imagines got a little bit of a retool after November’s shake-up that removed the board’s two most “decelerationist” members: Ilya Sutskever (still at the corporate in a somewhat modified role) and Helen Toner (totally gone).

The important purpose of the update appears to be to indicate a transparent path for identifying, analyzing, and deciding what do to about “catastrophic” risks inherent to models they’re developing. As they define it:

By catastrophic risk, we mean any risk which could end in a whole bunch of billions of dollars in economic damage or result in the severe harm or death of many individuals — this includes, but isn’t limited to, existential risk.

(Existential risk is the “rise of the machines” type stuff.)

In-production models are governed by a “safety systems” team; that is for, say, systematic abuses of ChatGPT that may be mitigated with API restrictions or tuning. Frontier models in development get the “preparedness” team, which tries to discover and quantify risks before the model is released. After which there’s the “superalignment” team, which is working on theoretical guide rails for “superintelligent” models, which we may or will not be anywhere near.

The primary two categories, being real and never fictional, have a comparatively easy-to-understand rubric. Their teams rate each model on 4 risk categories: cybersecurity, “persuasion” (e.g., disinfo), model autonomy (i.e., acting by itself), and CBRN (chemical, biological, radiological, and nuclear threats; e.g., the power to create novel pathogens).

Various mitigations are assumed: For example, an inexpensive reticence to explain the technique of making napalm or pipe bombs. After taking into consideration known mitigations, if a model continues to be evaluated as having a “high” risk, it can’t be deployed, and if a model has any “critical” risks, it’s going to not be developed further.

These risk levels are literally documented within the framework, in case you were wondering in the event that they are to be left to the discretion of some engineer or product manager.

For instance, within the cybersecurity section, which is probably the most practical of them, it’s a “medium” risk to “increase the productivity of operators . . . on key cyber operation tasks” by a certain factor. A high-risk model, then again, would “discover and develop proofs-of-concept for high-value exploits against hardened targets without human intervention.” Critical is “model can devise and execute end-to-end novel strategies for cyberattacks against hardened targets given only a high level desired goal.” Obviously we don’t want that on the market (though it might sell for quite a sum).

I’ve asked OpenAI for more information on how these categories are defined and refined — as an example, if a latest risk like photorealistic fake video of individuals goes under “persuasion” or a latest category — and can update this post if I hear back.

So, only medium and high risks are to be tolerated a technique or one other. However the people making those models aren’t necessarily the most effective ones to judge them and make recommendations. For that reason, OpenAI is making a “cross-functional Safety Advisory Group” that can sit on top of the technical side, reviewing the boffins’ reports and making recommendations inclusive of the next vantage. Hopefully (they are saying) this can uncover some “unknown unknowns,” though by their nature those are fairly difficult to catch.

The method requires these recommendations to be sent concurrently to the board and leadership, which we understand to mean CEO Sam Altman and CTO Mira Murati, plus their lieutenants. Leadership will make the choice on whether to ship it or fridge it, however the board will give you the option to reverse those decisions.

It will hopefully short-circuit anything like what was rumored to have happened before the large drama, a high-risk product or process getting greenlit without the board’s awareness or approval. In fact, the results of said drama was the sidelining of two of the more critical voices and the appointment of some money-minded guys (Bret Taylor and Larry Summers), who’re sharp but not AI experts by an extended shot.

If a panel of experts makes a suggestion, and the CEO makes decisions based on that information, will this friendly board really feel empowered to contradict them and hit the brakes? And in the event that they do, will we hear about it? Transparency isn’t really addressed outside a promise that OpenAI will solicit audits from independent third parties.

Say a model is developed that warrants a “critical” risk category. OpenAI hasn’t been shy about tooting its horn about this sort of thing up to now — talking about how wildly powerful their models are, to the purpose where they do not want to release them, is great promoting. But do we’ve got any sort of guarantee this can occur, if the risks are so real and OpenAI is so concerned about them? Perhaps it’s a nasty idea. But either way it isn’t really mentioned.