Probability & Information Theory Basics — Roadmap Step & Resources
Entropy, cross-entropy and KL divergence — the language most loss functions are written in
- Level: beginner
- Estimated time: 1 days
- Roadmap: Deep Learning & Neural Networks
Before this step
Interview questions
- Why is cross-entropy a natural loss function for classification?
Study resources
- Deep Learning Book: Probability and Information Theory (Article) — The free official Deep Learning book's chapter on the probability theory used throughout DL.
- Entropy (for data science) Clearly Explained!!! (Video) — StatQuest's widely-loved explanation of entropy, the foundation of cross-entropy loss.
Part of
- Mathematics for Deep Learning — section
- Deep Learning & Neural Networks roadmap — the full learning path