← Glossary
Glossary
What is Activation Function?
A function applied after each linear layer that introduces nonlinearity. Without it, stacking any number of linear layers collapses to a single linear transformation. ReLU, GELU, and SiLU are the most common. The choice directly affects whether gradients flow during training.
What people say
The nonlinear thing between layers