Binary entropy function

In information theory, the binary entropy function, denoted ${\displaystyle \operatorname {H} (p)}$ or ${\displaystyle \operatorname {H} _{\text{b}}(p)}$, is defined as the entropy of a Bernoulli process (i.i.d. binary variable) with probability ${\displaystyle p}$ of one of two values, and is given by the formula:

${\displaystyle \operatorname {H} (X)=-p\log p-(1-p)\log(1-p).}$

The base of the logarithm corresponds to the choice of units of information; base e corresponds to nats and is mathematically convenient, while base 2 (binary logarithm) corresponds to shannons and is conventional (as shown in the graph); explicitly:

${\displaystyle \operatorname {H} (X)=-p\log _{2}p-(1-p)\log _{2}(1-p).}$

Note that the values at 0 and 1 are given by the limit ${\displaystyle \textstyle 0\log 0:=\lim _{x\to 0^{+}}x\log x=0}$ (by L'Hôpital's rule); and that "binary" refers to two possible values for the variable, not the units of information.

When ${\displaystyle p=1/2}$, the binary entropy function attains its maximum value, 1 shannon (1 binary unit of information); this is the case of an unbiased coin flip. When ${\displaystyle p=0}$ or ${\displaystyle p=1}$, the binary entropy is 0 (in any units), corresponding to no information, since there is no uncertainty in the variable.