https://distill.pub/2020/circuits/weight-banding/
Weight Banding
Michael Petrov
Chelsea Voss
Ludwig Schubert
Nick Cammarata
Gabriel Goh
Chris Olah
April 8, 2021
DOI 10.23915/distill.00024.009
"Open up any ImageNet conv net and look at the weights in the last layer. You’ll find a uniform spatial pattern to them, dramatically unlike anything we see elsewhere in the network... the uniformity is so striking that when we first discovered it we thought it must be a bug.... We call this phenomenon weight banding."