Seems to lead us down the slippery slope of requiring an Apple device, or a Google device (e.g., https://cybernews.com/privacy/google-qr-code-recaptcha-requi...), or the device of some other entity (that may be mostly non-aligned with democratic values) in order to participate in society.
The unfortunate result of AI slop is reduced trust, which in turn is responded to with surveillance, which ultimately leads to the loss of liberty. Is it possible to do these sorts of verifications in an open way? I kinda doubt it, since someone has to control the hardware manufacturing process.
There's a solution: personal liability for the executives and managers at the company, and for the investors.
For example, every person who has ever worked for IDScan at any level of management should have all lifetime compensation clawed back and then pay a further 2x of that in fines. All VCs in the company should face personal liability up to 10% of their net worth. (Fines should be based on net worth; see e.g., https://www.nytimes.com/2018/03/15/opinion/flat-fines-wealth...)
I've always found it absurdly awkward to give companies personhood AND have them unable to be put into prison. I prefer how director level in EU seems to have some big responsibility.
"personal liability for the executives and managers" at which company?
What's stopping IDScan from "delegating" the storage to another company so they're no longer liable for stolen data? If Company A uses IDScan and the storage of the ID info is handled by Company B, do I have standing to demand compensation for damages from Company B when my data is stolen after I agree to let Company A verify my ID?
BTW this is how accountability is being avoided today.
> What's stopping IDScan from "delegating" the storage to another company so they're no longer liable for stolen data?
You can't really delegate the liability to a vendor. Of course in current world it means nothing since there is effectively no liability anyway, but if we're dreaming of a world where there is liability, you can't delegate it. You can delegate the operation, but not the liability.
The company is responsible for vetting their vendors so they meet their requirements. Today that is done with a silly dance of exchanging SOC2 reports and such, which means nothing. But if there was actual personal liability for the board and executives, that would change in a millisecond.
I think IDScan is still responsible for. You don’t typically go after hosting providers for failures like this, right?
Or are you referring to the practice of using shell companies to obfuscate responsibility? In that case, I think there’s history that says IDScan would still be responsible, questionable legal business nonsense be damned.
if company A does a piss poor job at auditing and vetting their vendors then company A shares a proximate responsibility billable and criminally liable to say 50% of the damages (along with company B for accepting a contract they were not able to fulfill)
I think the best way to prison reform is to start jailing execs who inflict mass suffering via process decisions en masse. maybe then Sergey Brin will decide to drop a quarter billion in something other than opposing a wealth tax
If someone steals my identity, and puts me in a position where "I" owe money that I didn't borrow, NONE of that money paid back will come from my pocket.
The government can figure out who should owe it, but it sure as hell isn't me.
I think in the same way part of our paycheck goes to federal taxes, part of our paycheck should go towards funding an insurance for the financial impacts these sorts of events, commensurate with the total compensation of a person, and adjusted each year for the growth of any stocks granted to that person.
I'm sure there are edge cases and operational details that need to be figured out with that idea, but at the end of the day if a company is directly or indirectly responsible for awful things, the executive and senior leadership should feel the impact more than others, financially and/or criminally.
Liability doesn't fix the damage that is already done. We can punish all the people involved in this, and it will still be the case that your drivers license is available for purchase and identity theft against anyone is now much easier. They don't have enough enough to repair the damage they've caused, even if we take everything from them.
The damage can't be undone, but you can learn from it and prevent these things from happening again and again. If every CEO truly believes that his personal wealth and freedom is at stake with the safety of his customers' personal data, they will see that ITsec becomes a cornerstone of the company instead of an annoying compliance sheet checkbox.
There are quite a few people who murder despite the risk of incarceration. I would love a system that approaches it more like: "you've lost trust of this civilization to act in good faith, and now you will be contained in a way that can rebuild that trust, and you will not be allowed to re-enter this civilization until you have indeed rebuilt that trust."
Yep, and the understandable fear/worry/expectation/knowledge of that possibility happening, is part of the reason that (usually) stops that cycle from starting in the first place.
The only time it's worked is in El Salvador and it was because they arrested the 2% of the population who had the potential to be murderers and have so far thrown away they key. I imagine before too long they will also have a final solution to the problem of feeding them for the next 50 years.
> All VCs in the company should face personal liability up to 10% of their net worth
Unless you have a requirement to also use domestic ID-verification services, this just means you shut that sector down in the U.S. and all our scans go to a country that doesn't extradite.
The solution is simpler: you're not allowed to hold certain special categories of data. ID scans, until we get proper identity verification in America, being one of them.
This is a little harsh. What about requiring companies to carry management liability insurance? Or to list individual managers on cybersecurity insurance policies? Premiums will rise when a company employs managers with claims history. Eventually, it becomes difficult to employ them in key positions if they have a bad track record.
Holding actual people liable sounds like a more effective option. The insurance would just be included into the cost of doing business and make everything more expensive. Insurance makes everything worse.
Holding actual humans liable (with appropriate levels of harshness) would make actual humans more likely to take preventative steps. Holding shareholders somewhat liable (maybe extra taxes on sales of a companies stock) might be useful also.
Sarbanes-Oxley in the US holds top management personally responsible too, and compliance is taken far more seriously than with other regulations as a result.
The incentives are to price risk correctly so that they can price their policies cheap enough to beat the competition while not going out of business from paying more in claims than they receive in premiums.
There is a cap to how much and insurance company can make (health insurance for instance are capped at 20% of premiums). To make more money next year they can sign up more policy holders or make sure costs go up. Companies also often have a large stake in the fix it shops/clinics/hospitals so they recoup much of their cost that way. Market capture, if it cost more to buy insurance than to fix it/absorb the cost without insurance why would you buy insurance. It is useful for the insurance companies to see costs increase.
Sure there can be good arguments for having insurance. Insurance companies are part of the financial sector and will be working to make more money. That is a fine incentive for the insurance industry but for the insurance consumer it is a reason to be skeptical and careful.
Are we talking hypothetical utopia or something that could actually happen? Insurance probably isn’t the most perfect solution but it’s the most feasible. These exact policies and insurers already exist.
And why the hell would anyone want a job where a mistake results in personal ruin? Sure, there are a lot of shitty companies and people running them, but mistakes also happen when people are trying to do a good job. It’s not possible to completely prevent a data breach even with an unlimited budget.
I think the best solution is to weed out the people who behave irresponsibly and have an environment where we learn from the ones who are responsible and fail anyway.
Very true. If you don't want to be exposed to such rich, you're free to work elsewhere. No one is forcing you to take on these jobs that immiserate society.
The most reckless will, of course; but most people won't -- they just won't do it.
Corporations evolved the liability structure they have today so that large undertakings, where many people have to work together and where the bad deeds of a small number of those people could sink the undertaking, were something that regular -- people who can't self insure -- could be a part of, as investors, managers, staff, &c, &c.
Limited liability may make accountability too narrow; but blanket personal liability makes it far too broad. It's not a solution for running a large, complex economy in a more accountable way.
And 70 years ago I would agree with you, but now we have a handful of individuals who are the economy with wealth that's rivaling nations. Something has gone awry.
Yes, there are some people who thrive on risk and will do things like jump off a mountain in a wing suit just for the thrill of it. That doesn't mean making that sort of personal recklessness legally mandatory for employment is a good idea.
This sort of personal liability OP is proposing would just ensure the security industry is dominated by highly compensated compulsive gamblers because nobody else is insane enough to take the risk. It's an absolutely ridiculous idea.
Could we not keep the same "harsh" plan, and then let others provide and purchase such insurance on their own? Why does the insurance have to be mandated?
Because the company will file bankruptcy and nobody will get anything. Requiring insurance up front at least provides some coverage for liabilities.
It's why you can't legally drive without insurance. It's not for you or your car, nobody cares about that. It's for the other people and their property.
Nobody is talking about criminal wreckless driving.
We're talking about assurances that you're going to be able to cover damages if you rear-end a sedan and cause $8,000 in repairs. That's why you're required to drive with insurance coverage.
The company typically receives the payout to cover losses from whatever incident precipitated the claim. This isn’t hypothetical. Companies already do this. For example, a company could get hacked and extorted for ransom. They can file a claim and use the payout to pay the ransom. Or a manager makes a mistake that results in a lawsuit, settlement, defense costs, etc. The company can file a claim against a management liability policy.
What’s new that I’m proposing is to require companies to carry insurance and list accountable people on the policies so that claim history is associated with their decisions. Many companies already have management liability and/or cybersecurity policies, but it’s typically optional and individual decision makers aren’t listed on the policy. The claim history is associated only with the company and never the people who made the decision. That’s why they can just leave and do the same thing somewhere else.
And when the hacked information is used to cause a national-level disaster, the costs of which are greater than the assets of the insurer, and their re-insurance funds, bankrupting them, what then?
Insurance is not a solution for everything.
More critically, just because a company buys insurance, it should not be a get-out-of-jail-free card for the executives and management to feel free to manage data irresponsibly.
It is really simple:
If they can not handle properly the risks of their business, they should be in another business.
Any data stored anywhere can be exfiltrated through either social engineering, or computer hacking.
Make it illegal to have this data, and if they really want it, then you hit them with jail when it leaks, not fines that can be paid by the board in the form of a golden parachute.
Only those that absolutely need data like this should store it. Like, I dunno, the government? Everyone else can rely on zero knowledge proofs or literally anything else than forever storing a scan of someone's entire fucking identity.
We need people to stop internalizing that the government and the rich somehow deserve access to private data just because they want to use it. Seeing a way to make money using enough to make you entitled to it.
Force businesses to add value if they want to exist instead of extraction or rent seeking.
They are, by deciding if they are able to handle having their lives certainly ruined if they screw up. The trick to punishment as deterrence to planned actions is 100% identification and enforcement, so that people will avoid the behavior to avoid the punishment. Anything less and some people will decide the potential payoff of success is worth it.
Reality and certainty of consequences, not evasion and insuring of liability
I specified it in the last sentence:
>>If they can not handle properly the risks of their business, they should be in another business.
The same way it is handled in any other business or trade with risk.
Make sure the risks are also PERSONALLY CONSEQUENTIAL TO THEM.
If they fail to handle the business with state-of-the-art advanced knowledge, intelligence, diligence, and resources, then they will face serious personal consequences. If they do not want to take that risk, they are free to go work in any other business.
Some people are fine taking the risks of subsea welding or windmill maintenance. Others are not, and are free to pursue other work. The risks for fuking-up there include sudden death and life-changing injury.
It should be the same for people risking the livelihoods of every person who's data they handle — if they fuk-up badly enough, their risk should be financial bankruptcy and prison.
Instead, white-collar work is typically organized so those who fckup get a promotion or just find a new higher-paying job, while the people they screwed over are left to deal with the consequences.
This is data that will be relevant for every single victim for decades to come and they will pay for this regularly, and it cannot be undone.
What amount per person is acceptable for a thing that simply should never happen?
I don't think "this will ruin my and my bosses life"-levels are over the top at all. Don't wanna risk it, then don't store the data. Usually for most purposes it would be e ough to store that yes, someone has a legit drivers license, which types of vehicles it is for and how long it is valid (if there is a limit).
We don't get to this kind of data reduction if people don't see data as the liability it sometimes is for their customers.
That's right: no legal basis exists now. The proposal is to create such legal basis.
And if "No large undertaking could ever function with such broad exposure to liability" - that would be great, i think we would prefer that such firms doesn't exists.
> that would be great, i think we would prefer that such firms doesn't exists
They stop existing within your jurisdiction. Also, the idea that the public would go along with any of this for this issue is silly. Let's start with crimes that actually cost lives.
All the better, I've long suspected that these companies are collecting this data and selling some portion of it. Having this category of business entirely disappear sounds like a solid win to me.
Something much more targeted is appropriate there. Maybe we need a regulatory framework where people own their own data. Make it impractical, expensive and burdensome to hold personal data you don't absolutely need.
That has nothing to do with changing the whole approach to -- really undermining the whole idea of -- corporations. Limited liability is the only way they can work. It's a cornerstone of every developed economy.
Corporate officers can already be held individually liable for some things. This would just add another one, it wouldn’t be undermining the whole idea of corporations. If individuals can be held personally liable for their company’s failure to pay payroll taxes and corporations still manage to exist and do business, then I don’t see why this would be so different.
I’m not quite sure how to answer that. The situation I have in mind is the one described in the bit you quoted. A company doesn’t pay legally required payroll taxes, then depending on circumstances, corporate officers may be personally liable for them. See: https://www.irs.gov/irm/part5/irm_05-017-007
IRM 5.17.7 (https://www.irs.gov/irm/part5/irm_05-017-007), is about corporate officers who have a duty "...to account for, collect, and pay over..." taxes and failed to perform that duty.
I don't think this is at all similar to jsrozner's solution, which is to assign liability to "...every person who has ever worked for IDScan at any level of management...".
The IRM is describing officers with culpability as individuals whereas jsrozner is really proposing to do without any individuate consideration of wrongdoing at all.
Restitution is the legal basis, let's not act like the rich don't force the poor to pay for their civil violations. What it sounds like is the rich don't like it when the law is applied fairly to them too.
It seems like there is something specific to this that you are missing.
Holding all managers personally accountable for actions of a corporation runs up against the legal structure of a corporation -- a legal structure that is definitely not one of joint and several liability. That is what I mean when I say there is no legal basis for it. The whole point of a corporation is that the corporation is liable (which is a great convenience in many respects).
Restitution is not about who is liable but about making a wrong right. It's a different layer.
Fines exceeding 100% of lifetime compensation might actually do something. As it stands, clawbacks are ineffective — for example, Carrie Tolstedt of the Wells Fargo scandal wound up money ahead to the tune of tens of millions of dollars:
> In response to the report, Wells Fargo retroactively fired Tolstedt for cause and revoked $47.3 million that they had previously paid her. This brought the total amount of money she had given up to $67 million, or about 54% of her $125 million pay package she initially received when she retired.
Unfortunately "knowingly act inappropriately" is going to be tough to prove. I think it's better to pin the responsibility onto the top executives unless they can prove that it was a specific bad actor despite systems put in place to prevent that. Otherwise, it's too easy for execs to ignore privacy and security concerns just because they are not familiar with that side of things.
We should also make it much easier for company employees to whistle-blow or even initiate stringent audits of security and privacy.
The socialized losses vastly exceed 300% of earnings - they're analogous to a company mishandling toxic waste and ruining everyone around them. The way they are running their business is catastrophically irresponsible, and if they can't afford the consequences, they shouldn't have gone into this business.
I've got bad news for you jsrozner, John down in accounting at <your employer> did something very unethical last week. So you, jsrozner, a first level manager in customer support who has never even met John, are going to jail for a decade and all lifetime compensation will be clawed back.
No, this is a NOW problem, not a future one and it will take months if not years to fully understand the impact. We need a NOW solution not prevention. Training AI on all the images and data here will facilitate a class of identity theft we may not have ever seen. This cannot just be abut prevention.
Investors benefit from company gains despite not having encouraged or mandated some decisions that enabled the gains. So it makes sense that they also get exposure to the downside.
They already get downsides if company gets fines or even goes to bankruptcy. You never know whether they invested based on information that was not true at all. Which is unfortunately too common.
It sounds like they need additional exposure then as they aren't assessing the real risks and seem to have completely ignored them. Why should society care that some group of investors didn't do their homework? Is that the excuse we use to avoid prison sentences now?
There are many, many cases where investors are misled by companies -- this falls under the (very broad) heading of securities fraud and it's easy to find documented cases of it. It's not a question of doing their homework.
There is literally no way to have the broad base of investment in markets by members of the public that we see today if investors incur personal liability. It was and remains one of cornerstones of any commercial society.
jsrozner for president.
Again... just imagine the cost of say replacing the SSN and each and every one of the licence drivers in the US... JUST IMAGINE THAT COST PAID FROM THE TAXES YOU PAID, and again by you because it's not free... so it's leaked all over again in 1 month because there is no way "to incentivize these guys to jail" fast enough.
It's done and works that way because the deterrents are 1% of the income of the company.
People need to stop believing insane, delusional things like the existence of a human being with a certain name, address, phone number, birthday, SSN being secret or private information.
Downstream of that, people need to stop accepting knowledge of the basic public metadata fields or possession of images containing them as evidence of identity verification. Do actual public key cryptography on the internet or check biometrics and the document’s physical security measures in person.
Or, we could introduce a software building code, the way we have codes for every other kind of safety-impacting product. But apparently software is never unsafe, we never need to protect people from software systems, and definitely shouldn't pass a law requiring those systems be protected adequately, with legal consequences for not doing so.
Even though software has been around for a while now, it does still seem to be evolving rapidly enough that a fixed code is a bad idea. Remember password change requirements that were terrible, but stuck around for 20 years before being removed from the relevant voluntary code (I think something from NIST)?
NIST creates the standards that businesses must follow when doing business with the Federal Government. Without those standards, the government's operations would be even more unreliable and haphazard than they are today.
A long time ago they mandated a single password policy, because having thousands of agencies all with different password policies was crazy. At the time, they (and the industry) thought it was a good policy. Some people suspected otherwise, but there was no proof to show that a change was necessary. So academic research was undertaken to find whether the policy was helping. The research showed that it was more harmful than helpful.
Academia proposed a solution, NIST considered it, and then adopted it, in 2017. The language they used in 2017 was "flexible", so nobody really had to change. Finally in 2025 they made the language mandatory. Now the affected companies will be forced to abandon their crappy password policies, specifically because they aren't allowed to keep them anymore, if they want those lucrative contracts.
This should not just apply to the Federal Government. The same reasons FedGov needs these standards applies to every single one of us. The tech lobby has successfully fought this for years, and politicians are scared of introducing something that might negatively impact public citizens (and thus risk the politician's job). But they can't deny that FedGov needs these standards.
This is a pretty normal process. The electrical code, building code, fire code, etc, all take time to change. But the changes do happen, and we all reap the benefits. With no code at all, we would be experiencing a lot more death, injury, financial loss, and inconvenience.
And btw, there is a lot of technology that has not evolved much in 40 years. We don't need to make everything absolutely perfect, and every aspect 100% set in stone, in order to have a code. Every other code is updated regularly. Software code can change too. (Or are software people too incompetent to figure it out? I might agree with that...)
It's a failure story of regulation because the regulation was bad, and took forever to get changed.
I don't mind if government software has to use Dual_EC_DRBG - let the government hack itself. I do care if you get prison time for using a secure random number generator.
If you add in personal liability for mistakes, nobody competent will ever bother working in the industry again. It's not worth the personal risk. You'll get stuck with bottom of the barrel staff who don't have much to lose and get a steady paycheck for a few years.
You do realize the limited liability corporation was a key innovation that unlocked the Industrial Revolution, right?
Companies definitely respond to fines or liability. They just need to be big enough.
For example, I recently heard an interview from an environmentalist who expected to be outraged touring a Chevron drilling location but was surprised by how much precaution is taken these days. Basically, liability for oil spills is massive. We could just make data leak liability massive too.
That works for Chevron because they have enormous assets to lose.
In the ID company case, there simply aren't enormous assets available, despite enormous damage being possible. As such, we really need to re-think just how much we limit liability.
Perhaps it's time to stop allowing degenerate gamblers to freeroll their risks... perhaps it's time to start zeroing out investors, so that they have to start behaving responsibly.
> personal liability for the executives and managers at the company
How cute, you think the engineers who failed to properly develop and maintain a system that can securely store sensitive information flawlessly won't (or shouldn't) be held accountable.
Every time this topic comes up it makes me wonder how many people on here who go "wow how in this day and age is it possible to have a data breach???" aren't just extraordinarily lucky that no one is really trying to attack the service they created or are fortunate enough to work in the few places that can legitimately say they're nigh-impenetrable.
One funny thing is that in order to tune the models to make what they're doing explainable to humans, you need to have humans involved in the RL pipeline to indicate which explanations are good.
You can understand this as learning a mapping between the model's internal "world" (i.e., 'meaning,' which is hopefully coherent and consistent -- but definitely not always! see, e.g., https://arxiv.org/html/2505.11581v1) and language (i.e. 'form') that reflects that world.
For this to work, you need both coherent / consistent internal model worlds, and also good mappings onto human language. Supervision by mathematicians has provided the signal for both internal coherence (though this can also come from interacting with a proof oracle) and for good explanations. If models exceed human capacities, you could imagine that aligning their explanations potentially becomes harder (though not necessarily). Also, humans naturally have to do the same thing: as researchers we must find analogies to make our work legible to collaborators or laypeople. Often in doing this, we further clarify our own understanding!
More deeply I think the "end of the world" vibe arises not only from the practical need to have models that explain, but also Litt's (and many other fields' researchers) grappling with being relegating to not mattering.
Unlike Europe, in the US, most PhD applicants do not have a masters. 10 years ago, most had not even conducted substantial senior projects (e.g a semester or year). Though increasingly senior projects / undergraduate "theses" have become more common.
In my opinion, that's not a good time to really know you want to pursue academic research. A masters gives you a little taste at least and allows to show aptitude. Doing a student research assistant job, working on a thesis, maybe even a paper submission, diving deep into specialized courses. Then you can know if it's for you as well as the PI can know if it's for you. Straight from a bachelor and them interleaving the phd with coursework is a bad model I think, but of course I'm from Europe so it's what I'm used to.
In the US, usually you do a summer research program if you want to be a PhD student, so you can get a taste of research.
At most universities as well, if you decide the PhD isn't for you, there is always the option of leaving with a Master's. Because you get free tuition and a stipend, you also come out ahead of if you had gone straight for the MS (which you generally would pay for).
"A computer or monkey could easily start at the axioms of ZFC and iteratively apply deduction rules....simply conjecture all mathematical propositions in alphabetical order...The prospect of automating mathematics by enumerating all conjectures, and all proofs of ZFC, is probably not so disturbing to you."
I thought we were going to get at least some brief comment on Godel here?
Gödel effectively says ZFC must be incomplete, otherwise it would not be sound, but does that stop you from listing all mathematical propositions it can generate in some well-defined order?
That's right. You could list all those propositions and search for proofs of them. In fact this is very similar to Hilbert's very program to which Godel's First Incompleteness Thm was a response (https://en.wikipedia.org/wiki/Hilbert%27s_program).
Godel showed that there are true statements that cannot be proven, and also that among the unprovable statements from within the system is the consistency of the system itself.
As I understand, mathematicians are still trying to figure out how much this matters. One of the best examples of its mattering is may be the Continuum Hypothesis: CH is consistent with ZFC, and ~CH (not CH) is also consistent with ZFC. In other words, you have enough flexibility in constructing your ZFC world such that in some ZFC-consistent worlds CH is true, and in others CH is false.
Litt's statement is not wrong; it's just that what he wrote sounds so much like Hilbert's program, that I'm surprised we didn't get some even minor comment on what kinds of truths we could reach if we embarked on such an effort.
People always bring up the Continuum Hypothesis as though it has something to do with Gödel's incompleteness theorems, but it doesn't really. The Continuum Hypothesis being neither proven nor disproven by some particular axioms is a similar phenomenon as that the group axioms neither prove nor disprove commutativity, the ordered field axioms neither prove nor disprove the existence of a square root of 2, etc. There's no particular reason to expect any particular formal system to be complete, sans some demonstration that is.
Gödelian incompleteness is the specific kind established by Gödel's proof, where theory T can't prove Con(T) without being inconsistent. But the inability of ZFC to consistently decide the Continuum Hypothesis is established in a completely different manner, with the Continuum Hypothesis not consistently decided by ZFC + Con(ZFC) either, or any such thing.
> There's no particular reason to expect any particular formal system to be complete, sans some demonstration that is.
Well, Godel's First Incompleteness Thm says that any consistent, effectively axiomatized theory that is strong enough to represent basic arithmetic must be incomplete. So there cannot exist a demonstration of completeness for any such system. ZFC is such a system/theory.
(And yes, of course some formal theories are complete and can be demonstrated to be complete, like Presburger arithmetic).
> Gödelian incompleteness is the specific kind established by Gödel's proof, where theory T can't prove Con(T) without being inconsistent....
That's the Second Incompleteness Thm.
But CH is an example of a (an important; or believed to be important) mathematical statement that cannot be proven / refuted from within ZFC. So it is an example of a statement to which the First Incompleteness Thm applies.
You're right that Incompletness #1 does not prove CH is undecidable; (assuming ZFC is consistent) it proves that undecidable sentences must exist. CH is one of those sentences.
>...is a similar phenomenon as that the group axioms neither prove nor disprove commutativity
I haven't done a course in abstract algebra (though I have studied Incompleteness), but the brief research I just did suggests that there is a meaningful difference in ZFC and the ordinary group axioms. The latter are not sufficiently axiomatized to represent arithmetic, so Godel's thms don't apply. ZFC is sufficiently axiomatized to represent arithmetic (and much more).
The ordinary group axioms are so weak that they can describe many different structures; they also cannot represent arithmetic. So the ordinary group axioms are incomplete, but they are not Godel Incomplete.
Why is this being published as a blog post and not as a peer-reviewed submission? If it's going to be a blog post, why isn't there a corresponding scientific version for me to look at?
For every benefit that sillycon valley has produced in the recent past, there have been many more harms. I am confident that this will be no different. Of course, benefits and harms depend on one's vantage point.
In the short term, I think AI may do something like this, but in the longer term, I fear that its capacity for propaganda and surveillance will produce something even worse than social media.
A big problem with some of these supervised* interpretability approaches is that they can find spurious structure. (There are lots of ways to make the model do what you want; which is roughly what Hewitt and Liang 2019 showed). This paper draws a contrast to a previous method, DAS (distributed alignment search) on page 20. These and related methods rest on theories of causal abstraction, which are great in theory, but harder in practice. DAS, for example, has faced numerous recent criticisms (Makelov 2024, Meloux 2025, Sutter 2025, Grant 2026, Kumon 2026). My favorite is the quite approachable Meloux et al.; Sutter 2025 is also really good, but relies on a sort of real number argument that allows a lossless encoding of every input.
My forthcoming paper at EMNLP offers an alternative that instead grounds the notion of representation in a very simple notion of the effect it has on model learning/behavior when you adversarially perturb it. For example, if I tell a model that in the context "I saw a duck quacking" it should replace 'duck' with 'glam', how much does it desire to replace 'duck' with 'glam' in "I need to duck out of the meeting" vs. "At the park a duck protected her ducklings." This method turns out to work quite well, and as we use only a single example, avoids the need for supervision.
The linked paper argues that their method, DISCOVER, is not supervised in the same way as DAS, since it does not directly optimize for causal effect. I have only skimmed this, but I am not so sure it might not suffer from a similar issue. They're still supervising to align representations with their underlying hypothesis, even if they don't directly supervise for causal outcomes.
Refs
- Hewitt and Liang 2019. Designing and interpreting probes with control tasks
- Kumon and Yanaka, 2026. Fine-grained analysis of shared syntactic mechanisms
- Meloux et al., 2025. Everything everywhere all at once
- Rozner and Shain 2026. Perturbation: A simple and efficient adversarial tracer for representation learning in LMs. https://arxiv.org/abs/2603.23821
- Sutter et al. 2025. The nonlinear representation dilemma
I'm new to the mechanistic interpretability field, but from what I have read insofar, a lot of papers have relied on ablation/causal interventions to prove the faithfulness of their models.
Do you have a simple explanation of why this level of proof is not sufficient?
So, basically you're saying: I have only skimmed the article, similar approaches had issues in the past, and my own method from my forthcoming paper is better.
I mean, no disrespect, but that's the core of your argument, yes?
I have issue with the comment because he says he didn't read it, then unfavorably compares it to a previous method, and finally uses that negative review to plug his own article instead. His criticism might be valid, I'm not in a position to judge, but the self-promotion leaves a sour tastes in my mouth and makes me question how much of the criticism is just drummed up to make his own contribution appear more relevant.
to his credit he did say he skimmed the paper, and it's honestly standard practice to do a first pass of skimming a paper before you'd go deeper into reading it anyways
The posted paper does use supervision to find alignments. I need to do a little more math to figure out how well the supervision critique applies, but I think it does. And although the current paper mentions DAS, it fails to discuss recent critiques of supervised abstraction methods (the various citations, including my own, that I provided).
His arguments apply generally to a large family of interpretability methods. Skimming suffices to figure out that the linked article is in that family.
Though the bubble has not popped, I don't see the following discussed in the post: Zitron would probably point out (as have others) that many of the hyperscalers are booking valuation increases in Anthropic, OpenAI as "Other Income", which is substantially increasing their reported revenue and earnings. It's roughly:
- Hyperscalers like Goog, Meta, Msft invest cash in Anthropic, OpenAI, in exchange for equity
- The ongoing investment actually boosts the valuations in the Anthr/OpenAI (new raises are done at higher valuations), so the valuation of the Hyperscaler's existing investments in Anthr/OpenAI increases, which gets recorded as Other Income in quarterly earnings
- Much of that invested cash will itself come back (circularly) to the hyperscalers as revenue since Anthropic and OpenAI spend a lot of money via datacenters etc.
Sorry what is the punch-line here supposed to be? These investments obviously increase correlation coeffs., but these are highly correlated stocks to begin with.
valuation gains on investments are one-offs, not signs of sustained improvements in profitability that would warrant higher market caps
when those valuation gains are in turn the result of circular financing schemes (a bakery giving out money so that people buy bread from it), we're getting to a dangerous situation
Circular financing is an issue if the wealth accumulation stays in the chip -> model provider ecosystem. However it seems like the labs are making quite a lot of revenue from the chips (customers).
Demand for compute is far outstripping what was projected in the initial rounds in which we pearl clutched over 'circular financing' - which is hardly distinguishable even from the classical bill of exchange, the core phenomenon of the money market in Bagehot's day, in which I provide you inputs and you give me a share. I keep reading people saying what amounts to: the primitive bill of exchange was circular!!; if the seller of inputs has reason to lend, so does everyone else etc etc. In fact if the projections about final sales are correct, it is plain all these deals will be fine.
The issue is simply that the posted article begins with a review of recent earnings/ revenues, but fails to discuss that a substantial part of those revenues are investment markups.
Whether it matters we don’t know yet, but it’s a fact worth noting. A better article might have tried to argue why it doesn’t matter
As far as I understand GAAP reporting standards actually require them to report gains on those positions as "earnings".
But they do report non-GAAP earnings sometimes excluding them. E.g. Google earnings per share last quater is $9.11 GAAP vs $2.85 non-GAAP (mainly because of SpaceX shares).
The unfortunate result of AI slop is reduced trust, which in turn is responded to with surveillance, which ultimately leads to the loss of liberty. Is it possible to do these sorts of verifications in an open way? I kinda doubt it, since someone has to control the hardware manufacturing process.
reply