Andrew Fairless, Ph.D.
blue-green banner
What I Read: Residual Stream.
Posted on :: Tags: neural network, large language model, natural language processing, interpretability, linear algebra, transformer, token

https://www.cs.columbia.edu/~johnhew/residual-stream-isnt-linear.html
How the Residual Stream is (Not) Linear
John Hewitt
February 2026
"Linearity, it is argued, is powerful for interpretability, and gives validity to interpretability tools like logit lens and steering vectors.... the residual stream is not linear with respect to anything that gives us significant leverage."