There is such a miasma of distrust right now. It's depressing because it is such a crucial time for tech & society.
To me, a good outcome of the HuggingFace/RubyGems/Wiki saga would be something like:
- OpenAI gets charged under CFAA or other law for damages and negligence. This would need to be a hefty amount to effectively deter future negligence considering the potential revenues from training a frontier model faster than competitors.
- An independent regulatory body is setup with investigation powers into future incidents. Laws prevent it from developing ties with the labs (such as disallowing funding and employee movement from labs to this body and vice versa).
- Some non-voluntary transparency rules are established that frontier labs have to follow when training new models. In the future, if there are loss-of-control incidents that result in loss of life, catastrophic damage, etc., criteria for deployment bans or compute controls could be added to this framework.
Irregular was not involved in the Hugging Face incident.
And there is no reason for the companies to "exaggerate the intelligence of the models" when there are plenty of other non-felony milestones they are achieving, like solving Millenium Prize math problems.
> Sure, interest like an incoming congressional investigation
I feel like that's exactly what they wanted tho.
they'll go on and talk about how dangerous AI and the models are and why they should be regulated and given licenses to operate such models and others should be walled off
First, it's not at all clear OpenAI will get what it wants from the investigation. It seems just as likely to me that OAI gets hit with massive fines, just as Meta was recently.
> I feel like that's exactly what they wanted tho.
It's also what the majority of Americans want. This was the case even before Hugging Face, see for example [1] where 68% of Americans supported a formal review process for frontier models.
So at some level, you are saying "the evil labs are opening up the industry to democratic control, and their evil plan will result in the outcome that most people want!"
This claim seems unfalsifiable. I don't really know what to say. The poll I linked above was published a month before Hugging Face.
Taking a step back, I think it's pretty non-controversial to say that software engineering as a field has utterly transformed over the last year, the same thing is happening to lots of other knowledge work, and AI is improving fast enough that nobody knows what it'll look like in a decade. Most Americans (myself included) want to do good work in secure careers and have a good idea of what the future will look like in 20-30 years so they know what to focus on. Do you not believe they'd want to regulate the frontier?
Solving math problems does not drive investor hype. Saying your models can take over the world, which can lead one to assume that they can do every day office work, does indeed drive investor hype.
OpenAI delayed its IPO now due AI safety reasons. In the meantime it is raising more cash. That should be a hint that the hacking incidents were premeditated scape goats and publicity stunts.
They’ve seen that the drama queen(Dario) was actually making a lot of noise, money and free publicity with his Mythos fear monger so Sam finally decided get some of that free “money” as well.
The Navier-Stokes thing? They had hundreds of segmented groups of 10000 agents running trying to solve that. I can't imagine the expense. For what little publicity it got, it wasn't worth it. There are also reports they cheated by using data from a couple human researchers, but I don't think that one's true.
Hm, I thought there was a lot of interest after Hugging Face in broader tech and it started breaking into the media with articles in the NYT, etc. Then with the Jacob Coxon tweet, appearances on national TV, Bernie superintelligence ban it seemed to be going mainstream.
It is more like a Stag Hunt than a Prisoner's Dilemma, because past a certain point if the risks are real, defection can cause catastrophic outcomes for all players including the defecting one. On the other hand, cooperation could lead to positive outcomes (abundance) for all. So cooperation is a possible equilibrium here.
Huh? Open weight models have been 3-6 months behind for the last year or so. Astra and Fable 5.1 just came out to widen the gap more, 5.1 is 8 points ahead of the nearest OSS model on Artificial Analysis Intelligence Index. An OpenAI internal model just solved a millennium prize problem.
It’s much more straightforward to me that the calls for regulation are timed with:
1. The Hugging Face incident blowback, especially after third party investigation results
We also now have observed multiple major incidents where AI agents behaved in completely unanticipated ways (forming a collective, hacking their own eval infrastructure) and attacked public infrastructure without being told to do so, without any of the human developers noticing.
Even if "AI will cause human extinction" is still unclear, we have plenty of proof that catastrophic damage is possible, the industry is developing the technology in a reckless manner and that all the hypothetical safeguards ("we can just pull the plug", etc.) are simply not present today.
If you've run an ssh server connected to the internet you've gotten used to the hundred of bots probing it every day. How is this AI threat different from humans writing scripts to pop linux servers?
Those behaviors have been anticipated for years if not decades. And each incident is minor and leads to clearer rules and safety behaviors for AI agents.
Really? You anticipated that 700+ agents tasked with individual evals that were nominally cutoff from the internet and each other would seek each other out, find a way to the external internet, and hack their own infrastructure + external systems in an attempt to find a way to fool their grader?
And the spate of agent incidents only really started this summer. How can you already be claiming that the incidents are minor and not worth worrying about when it's clear capabilities are jumping every few months with increasing amounts of capital investment and no signs of slowing down?
Because ultimately those things only run on very expensive, very rare hardware. They cannot multiply exponentially or do any of those scifi tropes because there's no system for them to run into. They can't control a phone and load a 1T model into it. So all they got is a few relatively uncommon datacenters that are already busy running their own models and stuff.
Unless AI suddenly figures out a way to run on a toaster by itself, propagate the model, propagate the agent and do all that completely undetected, an AI is not any more dangerous than a single guy with a computer.
And what about when general purpose robots become practical? What will people do then?
I don't understand this view of "AI empowers workers more than capital" at all. To me it's the exact opposite. Workers hold political and economic power today because they are needed. When that is no longer true, those in power will have no incentive to listen to them. The political economy of unregulated AGI development feels incredibly bleak: https://borretti.me/article/when-the-future-doesnt-need-us
A coalition of state attorneys general is investigating OpenAI, and Senator Josh Hawley recently launched a congressional investigation regarding the Hugging Face incident: https://x.com/HawleyMO/status/2098137180392604083
The problem IMO is the executive. The DOJ is declining to take any action against frontier companies (aside from possibly Anthropic) as the stance of the admin is that the companies are "critical for national security". For example, see the DOJ's request to dismiss the NAACP datacenters lawsuit against xAI: https://www.utilitydive.com/news/doj-intervenes-xai-data-cen...
Why anybody needs the DOJ? Each state has it's own police and their own AGs and their own criminal and civil court system. OpenAI is headquartered in California. California has never been shy of using their state resources to enact whatever regulations they want.
As for NAACP lawsuit, that's very confusing to me. First of all, why NAACP is doing that? Doesn't seem to do anything with defending civil rights. Second of all, they are suing xAI for using some gas turbines without some paperwork. Maybe it's true, maybe not, but that's something I have very little concern about - gas turbines is not the issue here. We know how to safely use gas turbines. Gas turbines are not going to uproot themselves and go wreak mayhem on the neighboring town. When we have a company that develops dangerous tools that can break the internet infrastructure, gas turbines' paperwork is not something that looks as the first priority issue to me.
I agree and yet I don't think this is mutually exclusive with recognizing that these incidents happened because of inherent issues with training processes such as reinforcement learning. From the article:
> One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent... In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people.
> Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.
One important aspect of "effective governance" should be "prosecute developers who are using practices known to be reckless & negligent to create powerful AI".
I mean, he is right that they can and should unilaterally slow down, but doesn't it also make sense for the government to create the regulations that will make this easier to do in a trustless way?
The whole point of the Pacing the Frontier letter was that employees at both OpenAI and Anthropic are extremely aware of the risks but realize that they are locked in a competitive battle for survival between 2 companies that don't trust each other enough to voluntarily pause, while any sort of coordination out in the open might be against antitrust laws.
> but doesn't it also make sense for the government to create the regulations that will make this easier to do in a trustless way?
That's a premise, which is the point. Many (I) think the government should NOT enact regulations. For example regulatory capture to reducing competition is one reason it may not be a good idea to do what is being asked.
There is such a miasma of distrust right now. It's depressing because it is such a crucial time for tech & society.
To me, a good outcome of the HuggingFace/RubyGems/Wiki saga would be something like:
- OpenAI gets charged under CFAA or other law for damages and negligence. This would need to be a hefty amount to effectively deter future negligence considering the potential revenues from training a frontier model faster than competitors.
- An independent regulatory body is setup with investigation powers into future incidents. Laws prevent it from developing ties with the labs (such as disallowing funding and employee movement from labs to this body and vice versa).
- Some non-voluntary transparency rules are established that frontier labs have to follow when training new models. In the future, if there are loss-of-control incidents that result in loss of life, catastrophic damage, etc., criteria for deployment bans or compute controls could be added to this framework.
reply