Small-scale study: 16 law professors were asked to judge short-answer Q&A practice materials written by one of the other 15 professors, or generated by an LLM (Gemini 2.5 Pro). They preferred the LLM materials in 75% of cases, with fairly strong inter-rater agreement. Has some…
Who's Who of AI
Mark J. Nelson
Why they matter
Researcher with public evidence across AI research.
- AI signals
- 16 past 30d
- Sources
- 12 distinct domains
- Discussions
- 26 past 30d
- Latest signal
- 1d ago
Comp. sci. prof. @ American University, Washington DC. AI & games researcher with miscellaneous other interests. https://www.kmjn.org/
What they're sharing
law.stanford.edu
Fine-Tuning Language Models to Know What They Know arxiv.org
google.com
View on Bluesky →
Who gets to understand AI? | Ai2 allenai.org
ft.com
Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies arxiv.org
CrankGPT — fully offline, human-powered local AI squeezlabs.github.io
China’s AI Boyfriend Business Is Taking On a Life of Its Own wired.com
compass.onlinelibrary.wiley.com
View on Bluesky →
Why AI’s Inference Boom Is Forcing a Rethink Of Chips and Memory spectrum.ieee.org
Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning arxiv.org
Mechanistic? aclanthology.org
Articles & links
Interesting looking new paper, "Fine-Tuning Language Models to Know What They Know", by Sangjun Park, Elliot Meyerson, Xin Qiu, Risto Miikkulainen (NeurIPS 2026). Proposes "a framework to measure and enhance LLM metacognition while controlling for... biases", w/ evolution stra…
AI Weekly's analysis
→
- A paper proposes ESMA (Evolution Strategy for Metacognitive Alignment), a method aimed at making LLMs accurately report what they do and don't know.
- The authors introduce a d'_type2 metric which, they say, isolates metacognitive ability from the biases and heuristics that confound existing evaluations.
- They report ESMA's gains generalize across unseen datasets, languages, and newly acquired knowledge, and trace to a sparse set of parameters.
Read full analysis →
Mark J. Nelson reposted
DeepMind is hiring a gameplay engineer in London. They have also hired an excellent narrative designer.
Mark J. Nelson reposted
Our fully open releases give researchers the data, code, checkpoints, and methods they need to inspect claims, reproduce findings, and advance new science. Read more about why that’s so important to us. ⬇️ allenai.org/blog/who-get...
AI Weekly's analysis
→
- Ai2 argues meaningful AI transparency requires not just open weights but training data, code, methods, checkpoints, evaluations, and documentation.
- Post cites three studies enabled by open Olmo releases, covering clinical demographic bias, benchmark inflation, and how models reason about drug names.
- Without that access, Ai2 warns, technical direction of the field risks becoming concentrated inside a small number of companies.
Read full analysis →
View on Bluesky →
An interesting development on the local-models front: one of the largest U.S. law firms is going to start fine-tuning and hosting its own LLMs in-house.
Some folks I know blame RL posttraining for this and are on a mission to show that ES posttraining doesn't produce the same problems arxiv.org/abs/2509.24372 arxiv.org/abs/2608.12679
AI Weekly's analysis
→
- The paper argues standard RL post-training narrows LLM output distributions and causes solution coverage to collapse on pass@k tests.
- Evolution Strategies, a population-based gradient-free method operating in weight space, achieved consistently higher pass@k than RL in the authors' tests.
- The abstract reports gains on standard math benchmarks but names no specific RL baseline, model sizes or benchmark figures.
Read full analysis →
finally some off-the-grid local AI
AI Weekly's analysis
→
- CrankGPT runs a full voice-interactive AI pipeline on a Raspberry Pi 5 with 8GB RAM, powered solely by a 20W hand-crank generator.
- Cold-start to functional conversation takes roughly 30 seconds; time to first token ranges from 0.8 to 2.9 seconds depending on model size.
- Memory bandwidth, not raw compute, is the primary bottleneck for on-device LLM inference, with DDR5 hardware achieving 29-58% faster token generation than DDR4.
Read full analysis →
did you see this one a few months ago?
Mark J. Nelson reposted
Now published in open access! Your one-stop shop for the philosophy of language models. It's the spiritual descendant of our two-part preprint from 2024, fully updated. This should be particularly useful for anyone looking for an entry point into this rapidly growing field.
Roundup by IEEE Spectrum of what's currently happening in specialized LLM inference chips.
AI Weekly's analysis
→
Read full analysis →
Some folks I know blame RL posttraining for this and are on a mission to show that ES posttraining doesn't produce the same problems arxiv.org/abs/2509.24372 arxiv.org/abs/2608.12679
There's at least one actually good paper's worth of methodological critique of mechinterp, but not from that faction, along pretty different lines, and not contemptuous.
Their own posts
Recent commentary
Underreported thing about Gemini (more than other LLMs I think) is that it's an ok replacement for Google Books, and fluidly multilingual. Like I can ask a question and request answers be based only on books by a specific academic (which are in Greek) and it will dig up relevant passages.
I like AI, but still, is this what I want to see at the Amtrak station
Noticing a pattern across departments that faculty are generally a lot more excited about AI than students, esp undergrad students.
Got a good 96gb-VRAM local LLM harness going and can't believe I used to just... put up with the garbage latencies that OpenAI/Anthropic/Google serve up.
Given general sentiment about AI-at-work out there, I assume this is not how many workers perceive things. But for general office work, I personally find using LLMs improves my enjoyment of the process *even when* it doesn't actually save any time.
The entanglement between the AI startup scene and AI academia is starting to produce some of the widespread conflict of interest problems you see in biomed imo.
There's a huge increase in single-author papers being submitted by early-career researchers in AI/ML. Leaving aside other issues, from a purely CV-maxxing perspective do these even help? I think on a hiring committee I'd give them mixed to negative weight in many cases.
A reason I've pulled back from reviewing for big AI conferences is a feeling that I'm doing unpaid supervision of other people's PhD students. Too many ppl submitting 10+ papers to a single conference where I doubt the prof whose name is on the paper has done a thorough review & revision themselves.
Not a strongly held opinion, but I'm a little skeptical of the recent LLM math results not really being compared against baseline search methods with similar compute budgets. Some of them are using pretty huge compute budgets!
Something I've started doing the past few semesters in Intro to AI that works pretty well (imo) is random neural policy search as the first "RL algorithm" we study. Has a bunch of nice features and provocations imo. Like: is this a "learning curve"? It has all the formal features of one!
Their network
In Mark J. Nelson's orbit
Center = Mark J. Nelson. Left = members they follow (green edges). Right = members who follow them (blue edges). Top = mutual follows (orange edges, slightly larger). Drag any node to reposition; click to open that profile.
Are you Mark J. Nelson? Show it.
Add the Who’s Who of AI badge to your site or bio. It links back to this profile.
Markdown: [](https://aiweekly.co/whos-who/person/mm-jj-nn-bsky-social)