Entropy

Entropy is a measure of the expected uncertainty (or information content) of a random variable.

$$H(X)=-\sum _{i=1}^{n}P(x_i){\log }_{2}P(x_i)$$

Where:

• $X$ is a random variable
• $x_i$ is the possible outcome
• $P(x_i)$ is the probability of the outcome $x_i$

Interpretation

• Entropy = 0 → no uncertainty (all outcomes are known)
• High entropy → high uncertainty (outcomes are mixed/unpredictable)