What is an Activation Function?

Detailed Deep Dive

Activation functions are mathematical formulas applied to the outputs of neural network nodes, dictating whether a node should activate and transmit signals forward. By introducing non-linearity, they allow deep neural networks to learn intricate, non-linear boundaries rather than just simple linear combinations of inputs. While historically functions like Sigmoid and Tanh were dominant, modern deep learning heavily relies on the Rectified Linear Unit (ReLU) and its variants (like Leaky ReLU or GeLU) because they mitigate the vanishing gradient problem and accelerate training.