← Timeline

Notes

Reactions

A lot of the hype around Nick’s post came from former OpenAI people and Goodfire investors. Most other people don’t know what’s happening and are just riding the wave of optimism.

Neel Nanda: interesting work, and evidence that neuron-based interpretability can work for practical problems, but not a big update to the big picture. He still thinks superposition is real. He and Nick had an interesting discussion in the replies.

Liv Gorton: respects Nick and would be happy if networks are easier to understand than we thought, but thinks the essay mischaracterises what superposition predicts.

Adam Majmudar calls it the most important result of the year, on the level of “Attention Is All You Need”, and says it takes us from “is alignment even solvable” to “we may solve alignment in the next year.”

It does look like a good result. Other researchers and scientists also put out healthy criticism.

Tarun Kathuria: some nice insights, but reading single neurons is about as useful as looking at one node and its neighbours to understand a graph with billions of nodes, and these graphs aren’t static.