And the third feature is that the empires monopolize knowledge production. So, within the last 10 years, we’ve seen the AI industry monopolize an increasing number of of the AI researchers on this planet. So AI researchers are not any longer contributing to open science, working in universities or independent institutions, and the effect on the research is what you’ll imagine would occur if a lot of the climate scientists on this planet were being bankrolled by oil and gas firms. You wouldn’t be getting a transparent picture, and we are usually not getting a transparent picture, of the restrictions of those technologies, or if there are higher ways to develop these technologies.
And the fourth and final feature is that empires at all times engage on this aggressive race rhetoric, where there are good empires and evil empires. And so they, the nice empire, should be strong enough to beat back the evil empire, and that’s the reason they need to have unfettered license to eat all of those resources and exploit all of this labor. And if the evil empire gets the technology first, humanity goes to hell. But when the nice empire gets the technology first, they’ll civilize the world, and humanity gets to go to heaven. So on many alternative levels, just like the empire theme, I felt prefer it was essentially the most comprehensive strategy to name exactly how these firms operate, and exactly what their impacts are on the world.
Niall Firth: Yeah, good. I mean, you talk in regards to the evil empire. What happens if the evil empire gets it first? And what I discussed at the highest is AGI. For me, it’s almost like the additional character within the book during. It’s form of looming over every part, just like the ghost on the feast, form of saying like, that is the thing that motivates every part at OpenAI. That is the thing we’ve got to get to before anyone else gets to it.
There’s a bit within the book about how they’re talking internally at OpenAI, like, we’ve got to be sure that that AGI is in US hands where it’s protected versus like anywhere else. And among the international staff are openly like—that’s form of a weird strategy to frame it, isn’t it? Why is the US version of AGI higher than others?
So tell us a bit about the way it drives what they do. And AGI isn’t an inevitable undeniable fact that’s just happening anyway, is it? It’s not even a thing yet.
Karen Hao: There’s not even consensus around whether or not it’s even possible or what it even is. There was recently a story by Cade Metz that was citing a survey of long-standing AI researchers in the sector, and 75% of them still think that we don’t have the techniques yet for reaching AGI, whatever meaning. And essentially the most classic definition or understanding of what AGI is, is having the ability to fully recreate human intelligence in software. But the issue is, we also don’t have scientific consensus around what human intelligence is. And so one in all the features that I speak about rather a lot within the book is that, when there may be a vacuum of shared meaning around this term, and what it could seem like, when would we now have arrived at it? What capabilities should we be evaluating these systems on to find out that we’ve gotten there? It could possibly principally just be whatever OpenAI wants.
So it’s form of just this ever-present goalpost that keeps shifting, depending on where the corporate desires to go. You understand, they’ve a full range, a wide range of different definitions that they’ve used throughout the years. In reality, they actually have a joke internally: For those who ask 13 OpenAI researchers what AGI is, you’ll get 15 definitions. In order that they are form of self-aware that this just isn’t really an actual term and it doesn’t really have that much meaning.
However it does serve this purpose of making a form of quasi-religious fervor around what they’re doing, where people think that they should keep driving towards this horizon, and that in the future once they get there, it’s going to have a civilizationally transformative impact. And due to this fact, what else must you be working on in your life, but this? And who else needs to be working on it, but you?
