I ran across an interesting post by Kevin Bass about Anthropic, AI safety, and the network of organizations involved in evaluating the risks posed by AI.
I don’t endorse everything in the post. In particular I don’t think it is necessary to assume corruption, conspiracy, or even bad faith to recognize the problem he identifies.
The problem is incentives.
Anthropic has been among the loudest voices warning that advanced AI may pose extraordinary risks. It has also advocated government regulation of frontier models, including independent evaluations intended to determine whether those models pose unacceptable risks.
Let’s assume they’re right.
Indeed, let’s go considerably farther than that. Let’s give Anthropic the keys.
If Anthropic really has the expertise to determine what constitutes a dangerous AI model, put it in charge of developing the standards. Give it an important role in evaluating models. Give its recommendations substantial regulatory force.
There should be one condition.
Anthropic can’t profit from it.
I don’t mean that figuratively. Congress has considerable power to establish the conditions under which governmental authority may be exercised. We already have conflict-of-interest laws, disclosure requirements, divestiture requirements, procurement rules, and restrictions intended to prevent people from using governmental authority for their own financial benefit.
Use them.
Construct whatever legal mechanism is necessary so that Anthropic, its principals, and those exercising the delegated authority cannot become richer because of the regulatory regime they devise. That might require divestiture. It might require some sort of regulated return. It might require a special corporate structure. Those are details for lawyers to work out.
The principle is simple enough: you can have the keys or you can have the money. You can’t have both.
There is a reason for doing that which has nothing to do with whether anyone at Anthropic is honest.
Regulation creates barriers to entry. The more elaborate the testing requirements, the more expensive compliance becomes. A company already spending billions of dollars developing frontier models is in a much better position to comply with a regulatory regime requiring billions of dollars than a prospective competitor is.
Consequently a company can advocate regulations for perfectly sincere reasons and still benefit enormously from them. The regulations may protect the public. They may also protect the incumbent.
Both things can be true at the same time.
That is why I don’t think accusations of corruption are particularly useful here. We don’t need to know what is in anybody’s heart. We can remove the conflict instead.
There is another advantage to this approach. It would provide a pretty good test of revealed preference.
If the people at Anthropic actually believe that frontier AI presents risks to human civilization sufficiently grave to justify extraordinary governmental intervention, asking them to surrender the opportunity to make extraordinary private profits from that intervention doesn’t seem unreasonable.
The greater the danger, the stronger the argument becomes.
Conversely, if the response is that giving up those profits would be intolerable, perhaps the situation is not quite as extraordinary as we have been told.
I don’t expect anything like this to happen. The major AI companies would oppose it. The investors would oppose it. Quite possibly the AI safety organizations would oppose it. There would be endless arguments that government could not afford to lose access to the expertise concentrated in the companies actually developing frontier AI.
My proposal doesn’t lose that expertise.
Use it.
Give Anthropic the keys.
Just don’t let them own the tollbooth.
A key point I didn’t mention in the post: there is nothing stopping Anthropic from throttling their pace of development. Or OpenAI. Or Alphabet.