Activation Functions — Roadmap Step & Resources
Sigmoid, tanh, ReLU, Leaky ReLU and softmax, and why the choice matters
- Level: intermediate
- Estimated time: 1 days
- Roadmap: Deep Learning & Neural Networks
Before this step
Interview questions
- Why has ReLU largely replaced sigmoid/tanh in hidden layers of modern networks?
Study resources
- Neural Networks Pt. 3: ReLU In Action!!! (Video) — StatQuest's widely-loved explanation of activation functions including ReLU.
- Activation Functions Explained (Video) — A clear comparison of sigmoid, tanh and ReLU activation functions.
- CampusX Deep Learning (Neural Networks, Activation Functions) — YouTube playlist (verified) (Video) — CampusX Deep Learning (Neural Networks, Activation Functions) — YouTube playlist (verified) — resource aligned with Activation Functions.
- 3Blue1Brown Neural Networks — YouTube playlist (verified) (Video) — 3Blue1Brown Neural Networks — YouTube playlist (verified) — resource aligned with Activation Functions.
Part of
- Neural Network Fundamentals — section
- Deep Learning & Neural Networks roadmap — the full learning path