Rectifier (neural networks)

In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function^[1][2] is an activation function defined as the non-negative part of its argument, i.e., the ramp function:

${\displaystyle \operatorname {ReLU} (x)=x^{+}=\max(0,x)={\frac {x+|x|}{2}}={\begin{cases}x&{\text{if }}x>0,\\0&x\leq 0\end{cases}}}$

where ${\displaystyle x}$ is the input to a neuron. This is analogous to half-wave rectification in electrical engineering.

ReLU is one of the most popular activation functions for artificial neural networks,^[3] and finds application in computer vision^[4] and speech recognition^[5][6] using deep neural nets and computational neuroscience.^[7][8]