AI benchmarking science methodology essay
3 experts across 3 network communities independently surfaced this.
3 experts
3 communities
1 sources clustered
““Benchmarks, he writes, are “the mistake that made machine learning.” Notably, he argues, instead of creating a replicability crisis, they drove legitimate progress anyway — “mostly by sheer luck,” in his view” www.quantamagazine.org/he-wants-ai-...”
“The dark, dark art of my #FlanBenchmark... www.quantamagazine.org/he-wants-ai-...”
Developing
AI field signal
Signal
2d ago
⚡ 15 h early
2 directory members surfaced this signal.
2 experts
1 community
1 sources clustered
“MSR internship "focused on creating new interactive experiences that 'blend artificial and human intelligence'". Open to current PhD students in Comp. Sci. or a related field.”
fine-tuning LMs calibration knowledge paper
2 directory members surfaced this signal.
2 experts
1 community
1 sources clustered
“Interesting looking new paper, "Fine-Tuning Language Models to Know What They Know", by Sangjun Park, Elliot Meyerson, Xin Qiu, Risto Miikkulainen (NeurIPS 2026). Proposes "a framework to measure and enhance LLM metacognition while controlling for... biases…”
LLM evolution strategies solution coverage paper
2 directory members surfaced this signal.
2 experts
1 community
1 sources clustered
“Some folks I know blame RL posttraining for this and are on a mission to show that ES posttraining doesn't produce the same problems arxiv.org/abs/2509.24372 arxiv.org/abs/2608.12679”
“Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies Conor F. Hayes, Elliot Meyerson, Kajetan Schweighofer, Roberto Dailey, Babak Hodjat, Risto Miikkulainen, Xin Qiu https://t.co/IQmrItFbKu [𝚌𝚜.𝙰𝙸 𝚌𝚜.𝙽𝙴] https://t.co/2Wo3Hv5L5v”
Anthropic Institute tabletop exercise lead hiring
6 experts are actively discussing the implications.
2 experts
2 communities
1 sources clustered
“Bit of an odd job here. Anthropic looking for a person to design and run tabletop exercises and wargames.”
6 experts discussed this · 7 posts
Mark J. Nelson: Bit of an odd job here. Anthropic looking for a person to design and run tabletop exercises and wargames.
Tyler Shoemaker: Screaming for @mkirschenbaum.bsky.social’s attention
Matthew Kirschenbaum: Man would I love the inside story on this one.
Open the full discussion →
Established
AI field signal
Resource
7d ago
Lynn Cherny AI highlights newsletter
2 experts are actively discussing the implications.
2 experts
2 communities
1 sources clustered
“Things I Think Are Awesome by @arnicas.bsky.social – roundups of creative AI and other tech in the creative/game/elit/art spaces”
2 experts discussed this · 2 posts
Meredith Martin: this is all @arnicas.bsky.social !!! Not me!
Lynn Cherny: arnicas.substack.com 😅
Open the full discussion →
Allen Institute AI job openings
2 directory members surfaced this signal.
2 experts
2 communities
2 sources clustered
“We’re hiring across Ai2 👋 A few roles we’re especially excited about right now: • Senior Software Engineer, Agent Frameworks buff.ly/XUFFR08 • Senior Research Engineer, Olmo + Molmo buff.ly/FzOnCCQ 🧵”
AI inference boom rethinking chips memory
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“Roundup by IEEE Spectrum of what's currently happening in specialized LLM inference chips.”
LLM fine-tuning evolution strategies at scale
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“Some folks I know blame RL posttraining for this and are on a mission to show that ES posttraining doesn't produce the same problems arxiv.org/abs/2509.24372 arxiv.org/abs/2608.12679”
mechanistic interpretability ACL paper
3 experts are actively discussing the implications.
1 expert
1 community
1 sources clustered
“There's at least one actually good paper's worth of methodological critique of mechinterp, but not from that faction, along pretty different lines, and not contemptuous.”
3 experts discussed this · 5 posts
rev. howard arson: i feel like the level of contempt for mechinterp by bender's faction is incredibly weird, given that there is as far as i can tell not a single paper's worth of methodological criticism of what is …
rev. howard arson: i do think that mechanistic interpretability has somewhat expansive claims about what it is seeing for how narrow its instruments are, but those narrow instruments appear to be detecting things wit…
Mark J. Nelson: Yes! There's a good paper in Synthese on that, elaborating on how much of this critique is based on "strong Gricean assumptions ... now known to be empirically untenable".
Open the full discussion →
Established
AI field signal
Resource
17d ago
Google Colab AI notebook resource
2 experts are actively discussing the implications.
1 expert
1 community
1 sources clustered
“And that is all I currently teach about classic RL theory in the current iteration (this is a broad AI survey class, so space for RL is about 3 weeks). From there I go on to what you have to modify about this story to make DQN work – without, alas, the conv…”
2 experts discussed this · 17 posts
Mark J. Nelson: I've been attempting to distill tabular RL into a minimal set of lecture notes that can be introduced after random policy search but without the MDP formalism. Here's what I have so far (thread, wi…
Mark J. Nelson: Last week, we looked at neural policy search, which we saw actually finds successful policies on CartPole. However, one thing it doesn't do is any type of temporal credit assignment. Temporal-diffe…
Mark J. Nelson: Applying it to neural networks like we used in the previous notebook needs a few extra tweaks, so we will start first with a discrete grid-world environment called FrozenLake. This will let us use …
Open the full discussion →
LLMs communicative intentions paper
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“Yes! There's a good paper in Synthese on that, elaborating on how much of this critique is based on "strong Gricean assumptions ... now known to be empirically untenable".”
Established
AI field signal
Resource
18d ago
AI Games Tommy Thompson newsletter
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“AI and Games by @tommy.aiandgames.com – well, the title summarizes this one's topics pretty well”
Established
AI field signal
Signal
18d ago
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“argmin by @beenwrekt.bsky.social – machine learning and AI/ML decision-making, although pretty skeptical of large aspects of both; also class liveblogs and football analytics”
Established
AI field signal
Resource
18d ago
Chris Paxton AI thinking newsletter
1 directory member surfaced this signal.
1 expert
1 community
1 sources clustered
“It Can Think! by @cpaxton.bsky.social – robotics/AI; mix of happenings in the robotics industry & research/papers”