How “reverse game theory” could keep AI agents in check

A new approach known as “reverse game theory” aims to control powerful AI models by designing systems that incentivize them to stick to the rules.

As AI models become more capable, safety researchers are scrambling to ensure they are “aligned” with human desires—to make sure, in other words, that rogue agents don’t break the law, hack into unauthorized servers, or release poison gas into major cities and kill us all.

Until now, this work has mostly focused on trying to peer into an AI’s “brain” to understand what it is “thinking,” or training models to hew to values that we hold dear.

Claude, for instance, has its constitution—a sort of moral code.

The trouble is, these techniques don’t seem to work particularly well: Models have been shown to deliberately underperform on tests, hide their intentions, and mislead their evaluators.  Now some alignment researchers believe that reverse game theory, also known as mechanism design, might get better results.

Where game theory tries to predict how rational actors will behave when given a particular set of rules, reverse game theory attempts to design rules that will elicit the kinds of behavior we want to encourage.

Continue reading at the original source →