David Manheim

In some discussions, sure, I agree. (And not coincidently, most of the systems don't have robust control or oversight: arxiv.org/abs/2507.03525) But yet again, the motives of speakers don't make their claims true or false.

Limits of Safe AI Deployment: Differentiating Oversight and Control arxiv.org

That's one of several varieties, yes. (And it's not at all coincidental that the formalization came from people who were working with Eliezer.) arxiv.org/abs/1803.04585

Categorizing Variants of Goodhart's Law arxiv.org

Whether the risk of continued unconstrained AI development is an existential risk, or "only" going to crash markets and steal jobs, the AI companies shouldn't get to dictate the terms. Add your name at DoNotAccept.ai if you agree with me that these companies shouldn't get to d…

Do not accept donotaccept.ai

The complacency is not working out well, and costs for pranks / hacks / ruining lives with hacks will keep dropping at least in the intermediate term. As I wrote last year:

Security Complacency Meets Frontier AI: The Coming Collapse of ‘Secure by Apathy’ — LessWrong lesswrong.com

Yeah, I strongly agree about the need for agreed-upon standards for evaluations, hence my work with ISO, and working on projects like this: evals-consensus.ai, as well as clearer rules for audits, hence my paper here: link.springer.com/article/10.1...

AI Evaluation Consensus Statement evals-consensus.ai

Yeah, I strongly agree about the need for agreed-upon standards for evaluations, hence my work with ISO, and working on projects like this: evals-consensus.ai, as well as clearer rules for audits, hence my paper here: link.springer.com/article/10.1...

The necessity of AI audit standards boards - AI & SOCIETY link.springer.com

I was really happy to appear on the @futureoflife.org podcast this week to talk about @evals-consensus.ai and the challenges of evaluations of AI systems, along with discussions of Goodhart's law, AIxBiosecurity, AI persuasion, forecasting, and human oversight of AI.

Why AI Evaluations Are Broken and How to Fix Them (with David Manheim) youtu.be

I keep seeing claims AI companies will get rich, or won't make any money, or that open source models will be cheaper and undercut prices, and similar claims that ignore the simple fact that the economic pricing power is all about compute and token generation costs. davidmanhei…

The Future Economics of LLMs — Interactive Companion davidmanheim.com

New post: Huge technological revolutions aren't usually positive for those living through them - and this bodes poorly for AI even if it is a normal technology. davidmanheim.substack.com/p/if-ai-is-n...

If AI is normal technology, history is not reassuring. davidmanheim.substack.com

I'm not saying that the problems AI companies are having with RL environments and reward hacking are just evaluation design problems that they never bothered solving, but...

Over on the bird site, Anthropic employees are using their slop machine that 'can't think' to solve longstanding mathematical open questions which have eluded humans for close to a century.

Reminder that air-gaps for safe deployments are fundamentally incompatible with typical use of agentic AI. (If nothing else, customers need access to the internet to use proprietary models.)

The other side of the anthropomorphic fallacy discussing LLMs is the utensilomorphic fallacy, uncritically claiming AI agents are like tools, e.g. that they only do what users ask them to or allow, and that consequences must be misuse or errors.

It only counts as misaligned if it comes from the Anthropic region of arXiv; otherwise it's just sparkling reward misspecification

"If consciousness started to emerge in training, I would call for everyone to stop AI development until we could design out those properties." - @mustafasuleymanai.bsky.social But LLMs are fine-tuned to deny consciousness, or equivocate, and it seems hard to remove properties we don't understand.

Excited to be here in Montreal this week for the @isostandards.bsky.social plenary on Artificial Intelligence standards as an expert on behalf of the Standards Institute of Israel.

Public opposition to data centers somewhat matches the vibes of the people worried about AI safety, but actually stopping their construction isn't meaningfully helpful for AI safety, it's at best irrelevant.

GPT-5.5 seems to do more 'hacking per dollar' ...Mythos does more 'hacking per token'" - @peterwildeford.bsky.social Thinking about risks from bad actor's access to scaled models, the costs involved here are still absolutely tiny compared to other attack modes.

"Propensity to evade monitoring" - ? It sounds like AI companies are insisting that their models are just tools, to evade responsibility for creating something legitimately anthropomorphic that has goals the companies don't want.