https://amaarora.github.io/posts/2024-07-04%20SWA.html
Sliding Window Attention, Longformer - The Long-Document Transformer
Aman Arora
July 4, 2024
“…we will look take a deep dive into Sliding Window Attention (SWA) that was introduced as part of the Longformer architecture…”