Self-Attention Mechanism — Roadmap Step & Resources
How every token attends to every other token to build contextual representations
- Level: advanced
- Estimated time: 2 days
- Roadmap: Deep Learning & Neural Networks
Interview questions
- What do the query, key and value vectors in self-attention each represent?
Study resources
- Attention for Neural Networks, Clearly Explained!!! (Video) — StatQuest's widely-loved explanation of attention, foundational to self-attention.
- The Illustrated Transformer (Article) — Jay Alammar's hugely popular, visual explanation of self-attention and Transformers.
Part of
- Transformers & Modern Architectures — section
- Deep Learning & Neural Networks roadmap — the full learning path